Encrypting an RDS DB instance – Step Functions version
Consider how to encrypt an unencrypted RDS DB instance.
The AWS official explanation is as follows
You can only encrypt an Amazon RDS DB instance when you create it, not after the DB instance is created.
However, because you can encrypt a copy of an unencrypted snapshot, you can effectively add encryption to an unencrypted DB instance. That is, you can create a snapshot of your DB instance, and then create an encrypted copy of that snapshot. You can then restore a DB instance from the encrypted snapshot, and thus you have an encrypted copy of your original DB instance.
Limitations of Amazon RDS encrypted DB instances
In this case, Step Functions will be used to achieve the above procedure.
Environment
Create a Step Functions state machine that does the following in order
- Create a snapshot of the unencrypted DB instance.
- Check the status of the created snapshot and confirm that it is “available”.
- Create a copy of the snapshot. Enable the encryption option when making the copy.
- Check the status of the created snapshot and confirm that it is “available”.
- Create a new DB instance from an encrypted snapshot.
During the procedure, the status of the snapshot is checked twice.
This is because the status of the snapshot immediately after creation is “creating”.
With this status, it is not possible to copy the snapshot or restore the DB instance.
Use the Choice and Wait states to address the above.
If the status is “available”, proceed to the next state.
If status is not “available”, move to Wait state and wait.
After waiting, check the status again.
The Lambda functions that make up the state machine use Python 3.8.
Create an EC2 instance.
Use to access the original or encrypted DB instance.
The OS for this instance is the latest Amazon Linux 2.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL
https://github.com/awstut-an-r/awstut-soa/tree/main/04/005
Explanation of key points of template files
Step Functions State Machine
Resources:
StateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
Definition:
Comment: !Sub "${Prefix}-StateMachine"
StartAt: CreateSnapshotState
States:
CreateSnapshotState:
Type: Task
Resource: !Ref FunctionArn1
Next: DescribeSnapshotState1
DescribeSnapshotState1:
Type: Task
Resource: !Ref FunctionArn2
Parameters:
snapshot_id.$: $.no_encrypted_snapshot_id
ResultPath: $.no_encrypted_snapshot_status
Next: ChoiceState1
ChoiceState1:
Type: Choice
Choices:
- Not:
Variable: $.no_encrypted_snapshot_status
StringEquals: !Ref SnapshotAvailableStatus
Next: WaitState1
- Variable: $.no_encrypted_snapshot_status
StringEquals: !Ref SnapshotAvailableStatus
Next: CopySnapshotState
WaitState1:
Type: Wait
Seconds: !Ref WaitSeconds
Next: DescribeSnapshotState1
CopySnapshotState:
Type: Task
Resource: !Ref FunctionArn3
Parameters:
instance_id.$: $.instance_id
snapshot_id.$: $.no_encrypted_snapshot_id
ResultPath: $.encrypted_snapshot_id
Next: DescribeSnapshotState2
DescribeSnapshotState2:
Type: Task
Resource: !Ref FunctionArn2
Parameters:
snapshot_id.$: $.encrypted_snapshot_id
ResultPath: $.encrypted_snapshot_status
Next: ChoiceState2
ChoiceState2:
Type: Choice
Choices:
- Not:
Variable: $.encrypted_snapshot_status
StringEquals: !Ref SnapshotAvailableStatus
Next: WaitState2
- Variable: $.encrypted_snapshot_status
StringEquals: !Ref SnapshotAvailableStatus
Next: RestoreDbInstanceState
WaitState2:
Type: Wait
Seconds: !Ref WaitSeconds
Next: DescribeSnapshotState2
RestoreDbInstanceState:
Type: Task
Resource: !Ref FunctionArn4
Parameters:
availability_zone.$: $.availability_zone
db_subnet_group_name.$: $.db_subnet_group_name
instance_id.$: $.instance_id
security_group_id.$: $.security_group_id
snapshot_id.$: $.encrypted_snapshot_id
ResultPath: $.encrypted_instance_id
End: true
LoggingConfiguration:
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt LogGroup.Arn
IncludeExecutionData: true
Level: ALL
RoleArn: !GetAtt StateMachineRole.Arn
StateMachineName: !Ref Prefix
StateMachineType: STANDARD
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "${Prefix}-StateMachineLogGroup"
StateMachineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- states.amazonaws.com
Policies:
- PolicyName: !Sub "${Prefix}-InvokeTaskFunctions"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- lambda:InvokeFunction
Resource:
- !Ref FunctionArn1
- !Ref FunctionArn2
- !Ref FunctionArn3
- !Ref FunctionArn4
- PolicyName: !Sub "${Prefix}-DeliverToCloudWatchLogPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutLogEvents
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: "*"
Code language: YAML (yaml)For more information on the basics of the Step Functions state machine, please see the following pages.
1st state (CreateSnapshotState)
This state creates a snapshot of the DB instance.
Set the Lambda function to execute in the Resources property.
In this state, specify the following Lambda function 1.
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
REGION: !Ref AWS::Region
Code:
ZipFile: |
import boto3
import os
region = os.environ['REGION']
client = boto3.client('rds', region_name=region)
def lambda_handler(event, context):
instance_id = event['instance_id']
response1 = client.describe_db_instances(
DBInstanceIdentifier=instance_id
)
response2 = client.create_db_snapshot(
DBSnapshotIdentifier='{instance}-no-encrypted'.format(instance=instance_id),
DBInstanceIdentifier=instance_id
)
return {
'instance_id': instance_id,
'availability_zone': response1['DBInstances'][0]['AvailabilityZone'],
'db_subnet_group_name': response1['DBInstances'][0]['DBSubnetGroup']['DBSubnetGroupName'],
'security_group_id': response1['DBInstances'][0]['VpcSecurityGroups'][0]['VpcSecurityGroupId'],
'no_encrypted_snapshot_id': response2['DBSnapshot']['DBSnapshotIdentifier'],
}
FunctionName: !Sub "${Prefix}-function1"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)This function takes an instance_id as an argument.
This is the ID of an unencrypted DB instance.
Run the describe_db_instances and create_db_snapshot methods.
The former retrieves information about the existing DB instance.
Specifically, the security group set for the DB instance, the DB subnet group, and the AZ where the instance is located.
The latter creates a snapshot of the DB instance.
As quoted at the beginning, this snapshot is not yet encrypted.
The DB instance information and the IDs of the snapshots created are compiled into a dictionary as the return value.
The state machine will retain these values.
For example, the following data would be retained.
{
"availability_zone": "ap-northeast-1d",
"instance_id": "soa-04-005",
"no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
"db_subnet_group_name": "dbsubnetgroup",
"security_group_id": "sg-09dfb82d21e4846c4"
}
Code language: JSON / JSON with Comments (json)2nd state (DescribeSnapshotState1)
This state checks the status of the snapshot created in the previous state.
Execute Lambda function 2.
The Parameter property allows you to specify parameters to be passed to the function.
For example, pass the following data.
{
"snapshot_id": "soa-04-005-no-encrypted"
}
Code language: JSON / JSON with Comments (json)The value of the data to be passed is referenced to the one held in the previous state.
Execute the following functions
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
REGION: !Ref AWS::Region
Code:
ZipFile: |
import boto3
import os
region = os.environ['REGION']
client = boto3.client('rds', region_name=region)
def lambda_handler(event, context):
snapshot_id = event['snapshot_id']
response = client.describe_db_snapshots(
DBSnapshotIdentifier=snapshot_id
)
return response['DBSnapshots'][0]['Status']
FunctionName: !Sub "${Prefix}-function2"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)Run the describe_db_snapshots method to check the status of the snapshots.
As we checked earlier, call the same method passing the snapshot ID as an argument.
Returns the obtained status information.
The ResultPath property of the state machine allows you to set how it receives the data passed from the function.
In this case, we store the status information in the key no_encrypted_snapshot_status.
In response, the data held by the state machine is updated.
The following is a concrete example.
{
"availability_zone": "ap-northeast-1d",
"instance_id": "soa-04-005",
"no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
"db_subnet_group_name": "dbsubnetgroup",
"security_group_id": "sg-09dfb82d21e4846c4",
"no_encrypted_snapshot_status": "creating"
}
Code language: JSON / JSON with Comments (json)3rd and 4th states (ChoiceState1, WaitState1)
These states work in conjunction with the previous state.
If the current snapshot status is “creating” or similar, wait for a while.
Check the status again and if it is “available”, go to the next state.
The above behavior is achieved with the StringEquals and Not properties of the Choice state and the Wait state.
For details on how to combine the Choice and Wait states to loop until a specific condition is met, see the following page.
This time, the conditional branching is based on the status of the created snapshot.
So we refer to the no_encrypted_snapshot_status that we just made the state machine hold.
5th state (CopySnapshotState)
This state copies the created snapshot with encryption enabled.
Execute Lambda function 3.
In the Parameter property, pass the following data, for example
{
"instance_id": "soa-04-005",
"snapshot_id": "soa-04-005-no-encrypted"
}
Code language: JSON / JSON with Comments (json)The ID of the aforementioned instance and the ID of the snapshot that has been created.
Execute the following functions
Resources:
Function3:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
REGION: !Ref AWS::Region
KMS_KEY_ID: !Sub "arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/rds"
Code:
ZipFile: |
import boto3
import os
region = os.environ['REGION']
kms_key_id = os.environ['KMS_KEY_ID']
client = boto3.client('rds', region_name=region)
def lambda_handler(event, context):
instance_id = event['instance_id']
snapshot_id = event['snapshot_id']
response = client.copy_db_snapshot(
SourceDBSnapshotIdentifier=snapshot_id,
TargetDBSnapshotIdentifier='{instance}-encrypted'.format(instance=instance_id),
KmsKeyId=kms_key_id,
)
return response['DBSnapshot']['DBSnapshotIdentifier']
FunctionName: !Sub "${Prefix}-function3"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)Use the copy_db_snapshot method to copy the snapshot.
The point is the argument KmsKeyId when calling this method.
Specifying this will activate encryption.
This parameter specifies the KMS key to be used for encryption.
In this case, we will use the AWS managed key for RDS.
Returns the ID of the snapshot that will be copied and newly created.
In the state machine, the key named encrypted_snapshot_id stores the snapshot ID.
In response, the data held by the state machine is updated.
The following is an example.
{
"availability_zone": "ap-northeast-1d",
"instance_id": "soa-04-005",
"no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
"db_subnet_group_name": "dbsubnetgroup",
"security_group_id": "sg-09dfb82d21e4846c4",
"no_encrypted_snapshot_status": "creating",
"encrypted_snapshot_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)6th through 8th states (DescribeSnapshotState2, ChoiceState2, WaitState2)
Same process as before.
A status check is performed on the snapshot with encryption enabled, and if “creating”, it is paused.
After suspending, another status check is performed and if “available”, proceed to the next state.
9th state (RestoreDbInstanceState)
This state creates a DB instance from an encrypted snapshot.
Execute Lambda function 4.
In the Parameter property, pass the following data, for example
{
"availability_zone": "ap-northeast-1d",
"instance_id": "soa-04-005",
"db_subnet_group_name": "dbsubnetgroup",
"security_group_id": "sg-09dfb82d21e4846c4",
"snapshot_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)The key point is the value of snapsho_id.
Specifying an encrypted snapshot ID here will create a DB instance from this.
Execute the following functions
Resources:
Function4:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
REGION: !Ref AWS::Region
Code:
ZipFile: |
import boto3
import os
region = os.environ['REGION']
client = boto3.client('rds', region_name=region)
def lambda_handler(event, context):
instance_id = event['instance_id']
snapshot_id = event['snapshot_id']
availability_zone = event['availability_zone']
db_subnet_group_name = event['db_subnet_group_name']
security_group_id = event['security_group_id']
response = client.restore_db_instance_from_db_snapshot(
DBInstanceIdentifier='{instance}-encrypted'.format(instance=instance_id),
DBSnapshotIdentifier=snapshot_id,
AvailabilityZone=availability_zone,
DBSubnetGroupName=db_subnet_group_name,
VpcSecurityGroupIds=[
security_group_id,
]
)
return response['DBInstance']['DBInstanceIdentifier']
FunctionName: !Sub "${Prefix}-function4"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)Execute the restore_db_instance_from_db_snapshot method to create a DB instance from the snapshot.
The return value is the ID of the DB instance created.
In the state machine, the key named encrypted_instance_id stores the ID of the DB instance.
In response, the data held by the state machine is updated.
The following is an example.
{
"availability_zone": "ap-northeast-1d",
"instance_id": "soa-04-005",
"no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
"db_subnet_group_name": "dbsubnetgroup",
"security_group_id": "sg-09dfb82d21e4846c4",
"no_encrypted_snapshot_status": "creating",
"encrypted_snapshot_id": "soa-04-005-encrypted",
"encrypted_instance_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)(Reference) RDS
Resources:
DBInstance:
Type: AWS::RDS::DBInstance
DeletionPolicy: Delete
Properties:
AllocatedStorage: !Ref DBAllocatedStorage
AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone}"
DBInstanceClass: !Ref DBInstanceClass
DBInstanceIdentifier: !Ref Prefix
DBSubnetGroupName: !Ref DBSubnetGroup
Engine: !Ref DBEngine
EngineVersion: !Ref DBEngineVersion
MasterUsername: !Ref DBMasterUsername
MasterUserPassword: !Ref DBMasterUserPassword
StorageEncrypted: false
VPCSecurityGroups:
- !Ref DBSecurityGroup
DBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupName: dbsubnetgroup
DBSubnetGroupDescription: testgroup.
SubnetIds:
- !Ref DBSubnet1
- !Ref DBSubnet2
Code language: YAML (yaml)No special configuration is performed.
Create a DB instance.
(Reference) EC2
Resources:
Instance:
Type: AWS::EC2::Instance
Properties:
IamInstanceProfile: !Ref InstanceProfile
ImageId: !Ref ImageId
InstanceType: !Ref InstanceType
NetworkInterfaces:
- DeviceIndex: 0
SubnetId: !Ref InstanceSubnet
GroupSet:
- !Ref InstanceSecurityGroup
UserData: !Base64 |
#!/bin/bash -xe
yum update -y
yum install -y mariadb
Code language: YAML (yaml)User data defines the initialization process.
For more information on user data, see the following page.
Prepare a client for DB connection by installing MariaDB.
For more information, please see the following page.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormations stack and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- EC2 instance: i-0c948c4d3fa569462
- DB Instance: soa-04-005
- DB instance endpoint: soa-04-005.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com
- Lambda function 1: soa-04-005-function1
- Lambda function 2: soa-04-005-function2
- Lambda function 3: soa-04-005-function3
- Lambda function 4: soa-04-005-function4
- Step Functions state machine: soa-04-005
Check the DB instance from the AWS Management Console.
Storage shows that Encryption is “Not enabled”.
In other words, it is not encrypted.
Check the Step Functions state machine.
Indeed, a state machine has been created.
The flow of the state machine is represented as a graph.
Operation Check
Connect to unencrypted DB instance
Now that you are ready, access the EC2 instance.
SSM Session Manager is used to access EC2 instances.
% aws ssm start-session --target i-0c948c4d3fa569462
...
sh-4.2$
Code language: Bash (bash)For more information on SSM Session Manager, please refer to the following page.
Check the installation status of the MySQL client.
sh-4.2$ mysql -V
mysql Ver 15.1 Distrib 5.5.68-MariaDB, for Linux (aarch64) using readline 5.1
Code language: Bash (bash)Indeed, the MySQL client is installed by user data.
Use this client package to connect to the DB instance.
sh-4.2$ mysql -h soa-04-005.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com -P 3306 -u testuser -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 8.0.28 Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>
Code language: Bash (bash)Connection made.
Create a test database and tables to store test data.
MySQL [(none)]> CREATE database test;
Query OK, 1 row affected (0.01 sec)
MySQL [(none)]> use test;
Database changed
MySQL [test]> CREATE TABLE planet (id INT UNSIGNED AUTO_INCREMENT, name VARCHAR(30), PRIMARY KEY(id));
Query OK, 0 rows affected (0.02 sec)
MySQL [test]> INSERT INTO planet (name) VALUES ("Mercury");
Query OK, 1 row affected (0.00 sec)
MySQL [test]> select * from planet;
+----+---------+
| id | name |
+----+---------+
| 1 | Mercury |
+----+---------+
1 row in set (0.00 sec)
Code language: Bash (bash)Test data could be written.
Step Functions state machine execution
Run the state machine.
Set arguments to Input.
The argument is the ID of the unencrypted DB instance.
Press Start execution.
State machine execution has started.
Now each state will be executed in turn.
Paused at WaitState1.
Input shows that the value of no_encypted_snapshot_status is “creating”.
In other words, we are waiting for the status of the snapshot created from the DB instance to change to “available”.
Check the details of this snapshot.
Looking at the Status, it is indeed “creating”.
In addition, the KMS key ID is not set, so it is not encrypted.
After a short wait, the process moves to the next state.
Paused at WaitState2.
If you look at the Input, you will see that the value of encypted_snapshot_status is “creating”.
This snapshot was created by copying the previous snapshot with the encryption option enabled.
We are waiting for the status of this to change to “available”.
Check the details of this snapshot.
Looking at the Status, it is indeed “creating”.
In addition, the KMS key ID is set, so it is encrypted.
After waiting for a while, processing of all states is completed.
A new DB instance has been created from the encrypted snapshot.
The ID of the DB instance is “soa-04-005-encrypted”.
Encrypted DB instance
Check the newly created DB instance.
Encryption shows “enabled”.
This indicates that it is indeed encrypted.
Finally, access this DB instance.
sh-4.2$ mysql -h soa-04-005-encrypted.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com -P 3306 -u testuser -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.28 Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>
Code language: Bash (bash)I was able to log in successfully.
Check the stored data.
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.00 sec)
MySQL [(none)]> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MySQL [test]>
MySQL [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| planet |
+----------------+
1 row in set (0.01 sec)
MySQL [test]> select * from planet;
+----+---------+
| id | name |
+----+---------+
| 1 | Mercury |
+----+---------+
1 row in set (0.00 sec)
Code language: Bash (bash)Data was saved.
This was saved before encryption.
From the above, we have created an encrypted DB instance with the same contents as the unencrypted DB instance.
Summary
We have shown how to encrypt an unencrypted RDS DB instance using Step Functions.