SOA_EN

Encrypting an RDS DB instance – Step Functions version

Encrypting an RDS DB instance – Step Functions version

Consider how to encrypt an unencrypted RDS DB instance.

The AWS official explanation is as follows

You can only encrypt an Amazon RDS DB instance when you create it, not after the DB instance is created.

However, because you can encrypt a copy of an unencrypted snapshot, you can effectively add encryption to an unencrypted DB instance. That is, you can create a snapshot of your DB instance, and then create an encrypted copy of that snapshot. You can then restore a DB instance from the encrypted snapshot, and thus you have an encrypted copy of your original DB instance.

Limitations of Amazon RDS encrypted DB instances

In this case, Step Functions will be used to achieve the above procedure.

Environment

Diagram of encrypting an RDS DB instance - Step Functions version.

Create a Step Functions state machine that does the following in order

  1. Create a snapshot of the unencrypted DB instance.
  2. Check the status of the created snapshot and confirm that it is “available”.
  3. Create a copy of the snapshot. Enable the encryption option when making the copy.
  4. Check the status of the created snapshot and confirm that it is “available”.
  5. Create a new DB instance from an encrypted snapshot.

During the procedure, the status of the snapshot is checked twice.
This is because the status of the snapshot immediately after creation is “creating”.
With this status, it is not possible to copy the snapshot or restore the DB instance.

Use the Choice and Wait states to address the above.
If the status is “available”, proceed to the next state.
If status is not “available”, move to Wait state and wait.
After waiting, check the status again.

The Lambda functions that make up the state machine use Python 3.8.

Create an EC2 instance.
Use to access the original or encrypted DB instance.
The OS for this instance is the latest Amazon Linux 2.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL

awstut-soa/04/005 at main · awstut-an-r/awstut-soa
Contribute to awstut-an-r/awstut-soa development by creating an account on GitHub.

Explanation of key points of template files

Step Functions State Machine

Resources:
  StateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      Definition:
        Comment: !Sub "${Prefix}-StateMachine"
        StartAt: CreateSnapshotState
        States:
          CreateSnapshotState:
            Type: Task
            Resource: !Ref FunctionArn1
            Next: DescribeSnapshotState1
          DescribeSnapshotState1:
            Type: Task
            Resource: !Ref FunctionArn2
            Parameters:
              snapshot_id.$: $.no_encrypted_snapshot_id
            ResultPath: $.no_encrypted_snapshot_status
            Next: ChoiceState1
          ChoiceState1:
            Type: Choice
            Choices:
              - Not:
                  Variable: $.no_encrypted_snapshot_status
                  StringEquals: !Ref SnapshotAvailableStatus
                Next: WaitState1
              - Variable: $.no_encrypted_snapshot_status
                StringEquals: !Ref SnapshotAvailableStatus
                Next: CopySnapshotState
          WaitState1:
            Type: Wait
            Seconds: !Ref WaitSeconds
            Next: DescribeSnapshotState1
          CopySnapshotState:
            Type: Task
            Resource: !Ref FunctionArn3
            Parameters:
              instance_id.$: $.instance_id
              snapshot_id.$: $.no_encrypted_snapshot_id
            ResultPath: $.encrypted_snapshot_id
            Next: DescribeSnapshotState2
          DescribeSnapshotState2:
            Type: Task
            Resource: !Ref FunctionArn2
            Parameters:
              snapshot_id.$: $.encrypted_snapshot_id
            ResultPath: $.encrypted_snapshot_status
            Next: ChoiceState2
          ChoiceState2:
            Type: Choice
            Choices:
              - Not:
                  Variable: $.encrypted_snapshot_status
                  StringEquals: !Ref SnapshotAvailableStatus
                Next: WaitState2
              - Variable: $.encrypted_snapshot_status
                StringEquals: !Ref SnapshotAvailableStatus
                Next: RestoreDbInstanceState
          WaitState2:
            Type: Wait
            Seconds: !Ref WaitSeconds
            Next: DescribeSnapshotState2
          RestoreDbInstanceState:
            Type: Task
            Resource: !Ref FunctionArn4
            Parameters:
              availability_zone.$: $.availability_zone
              db_subnet_group_name.$: $.db_subnet_group_name
              instance_id.$: $.instance_id
              security_group_id.$: $.security_group_id
              snapshot_id.$: $.encrypted_snapshot_id
            ResultPath: $.encrypted_instance_id
            End: true
      LoggingConfiguration:
        Destinations:
          - CloudWatchLogsLogGroup:
              LogGroupArn: !GetAtt LogGroup.Arn
        IncludeExecutionData: true
        Level: ALL
      RoleArn: !GetAtt StateMachineRole.Arn
      StateMachineName: !Ref Prefix
      StateMachineType: STANDARD

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties: 
      LogGroupName: !Sub "${Prefix}-StateMachineLogGroup"
      
  StateMachineRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - states.amazonaws.com
      Policies:
        - PolicyName: !Sub "${Prefix}-InvokeTaskFunctions"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - lambda:InvokeFunction
                Resource:
                  - !Ref FunctionArn1
                  - !Ref FunctionArn2
                  - !Ref FunctionArn3
                  - !Ref FunctionArn4
        - PolicyName: !Sub "${Prefix}-DeliverToCloudWatchLogPolicy"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogDelivery
                  - logs:GetLogDelivery
                  - logs:UpdateLogDelivery
                  - logs:DeleteLogDelivery
                  - logs:ListLogDeliveries
                  - logs:PutLogEvents
                  - logs:PutResourcePolicy
                  - logs:DescribeResourcePolicies
                  - logs:DescribeLogGroups
                Resource: "*"
Code language: YAML (yaml)

For more information on the basics of the Step Functions state machine, please see the following pages.

1st state (CreateSnapshotState)

This state creates a snapshot of the DB instance.

Set the Lambda function to execute in the Resources property.
In this state, specify the following Lambda function 1.

Resources:
  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          import boto3
          import os
          
          region = os.environ['REGION']
          
          client = boto3.client('rds', region_name=region)
          
          def lambda_handler(event, context):
            instance_id = event['instance_id']
            
            response1 = client.describe_db_instances(
              DBInstanceIdentifier=instance_id
            )
            
            response2 = client.create_db_snapshot(
              DBSnapshotIdentifier='{instance}-no-encrypted'.format(instance=instance_id),
              DBInstanceIdentifier=instance_id
            )
            
            return {
              'instance_id': instance_id,
              'availability_zone': response1['DBInstances'][0]['AvailabilityZone'],
              'db_subnet_group_name': response1['DBInstances'][0]['DBSubnetGroup']['DBSubnetGroupName'],
              'security_group_id': response1['DBInstances'][0]['VpcSecurityGroups'][0]['VpcSecurityGroupId'],
              'no_encrypted_snapshot_id': response2['DBSnapshot']['DBSnapshotIdentifier'],
            }
      FunctionName: !Sub "${Prefix}-function1"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

This function takes an instance_id as an argument.
This is the ID of an unencrypted DB instance.

Run the describe_db_instances and create_db_snapshot methods.

The former retrieves information about the existing DB instance.
Specifically, the security group set for the DB instance, the DB subnet group, and the AZ where the instance is located.

The latter creates a snapshot of the DB instance.
As quoted at the beginning, this snapshot is not yet encrypted.

The DB instance information and the IDs of the snapshots created are compiled into a dictionary as the return value.
The state machine will retain these values.
For example, the following data would be retained.

{
  "availability_zone": "ap-northeast-1d",
  "instance_id": "soa-04-005",
  "no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
  "db_subnet_group_name": "dbsubnetgroup",
  "security_group_id": "sg-09dfb82d21e4846c4"
}
Code language: JSON / JSON with Comments (json)

2nd state (DescribeSnapshotState1)

This state checks the status of the snapshot created in the previous state.
Execute Lambda function 2.

The Parameter property allows you to specify parameters to be passed to the function.
For example, pass the following data.

{
  "snapshot_id": "soa-04-005-no-encrypted"
}
Code language: JSON / JSON with Comments (json)

The value of the data to be passed is referenced to the one held in the previous state.

Execute the following functions

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          import boto3
          import os
          
          region = os.environ['REGION']
          
          client = boto3.client('rds', region_name=region)
          
          def lambda_handler(event, context):
            snapshot_id = event['snapshot_id']
            
            response = client.describe_db_snapshots(
              DBSnapshotIdentifier=snapshot_id
            )

            return response['DBSnapshots'][0]['Status']
      FunctionName: !Sub "${Prefix}-function2"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

Run the describe_db_snapshots method to check the status of the snapshots.
As we checked earlier, call the same method passing the snapshot ID as an argument.
Returns the obtained status information.

The ResultPath property of the state machine allows you to set how it receives the data passed from the function.
In this case, we store the status information in the key no_encrypted_snapshot_status.

In response, the data held by the state machine is updated.
The following is a concrete example.

{
  "availability_zone": "ap-northeast-1d",
  "instance_id": "soa-04-005",
  "no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
  "db_subnet_group_name": "dbsubnetgroup",
  "security_group_id": "sg-09dfb82d21e4846c4",
  "no_encrypted_snapshot_status": "creating"
}
Code language: JSON / JSON with Comments (json)

3rd and 4th states (ChoiceState1, WaitState1)

These states work in conjunction with the previous state.
If the current snapshot status is “creating” or similar, wait for a while.
Check the status again and if it is “available”, go to the next state.

The above behavior is achieved with the StringEquals and Not properties of the Choice state and the Wait state.
For details on how to combine the Choice and Wait states to loop until a specific condition is met, see the following page.

This time, the conditional branching is based on the status of the created snapshot.
So we refer to the no_encrypted_snapshot_status that we just made the state machine hold.

5th state (CopySnapshotState)

This state copies the created snapshot with encryption enabled.
Execute Lambda function 3.

In the Parameter property, pass the following data, for example

{
  "instance_id": "soa-04-005",
  "snapshot_id": "soa-04-005-no-encrypted"
}
Code language: JSON / JSON with Comments (json)

The ID of the aforementioned instance and the ID of the snapshot that has been created.

Execute the following functions

Resources:
  Function3:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          REGION: !Ref AWS::Region
          KMS_KEY_ID: !Sub "arn:aws:kms:${AWS::Region}:${AWS::AccountId}:alias/aws/rds"
      Code:
        ZipFile: |
          import boto3
          import os
          
          region = os.environ['REGION']
          kms_key_id = os.environ['KMS_KEY_ID']
          
          client = boto3.client('rds', region_name=region)
          
          def lambda_handler(event, context):
            instance_id = event['instance_id']
            snapshot_id = event['snapshot_id']
            
            response = client.copy_db_snapshot(
              SourceDBSnapshotIdentifier=snapshot_id,
              TargetDBSnapshotIdentifier='{instance}-encrypted'.format(instance=instance_id),
              KmsKeyId=kms_key_id,
            )
            
            return response['DBSnapshot']['DBSnapshotIdentifier']
      FunctionName: !Sub "${Prefix}-function3"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

Use the copy_db_snapshot method to copy the snapshot.
The point is the argument KmsKeyId when calling this method.
Specifying this will activate encryption.
This parameter specifies the KMS key to be used for encryption.
In this case, we will use the AWS managed key for RDS.

Returns the ID of the snapshot that will be copied and newly created.

In the state machine, the key named encrypted_snapshot_id stores the snapshot ID.
In response, the data held by the state machine is updated.
The following is an example.

{
  "availability_zone": "ap-northeast-1d",
  "instance_id": "soa-04-005",
  "no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
  "db_subnet_group_name": "dbsubnetgroup",
  "security_group_id": "sg-09dfb82d21e4846c4",
  "no_encrypted_snapshot_status": "creating",
  "encrypted_snapshot_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)

6th through 8th states (DescribeSnapshotState2, ChoiceState2, WaitState2)

Same process as before.
A status check is performed on the snapshot with encryption enabled, and if “creating”, it is paused.
After suspending, another status check is performed and if “available”, proceed to the next state.

9th state (RestoreDbInstanceState)

This state creates a DB instance from an encrypted snapshot.
Execute Lambda function 4.

In the Parameter property, pass the following data, for example

{
  "availability_zone": "ap-northeast-1d",
  "instance_id": "soa-04-005",
  "db_subnet_group_name": "dbsubnetgroup",
  "security_group_id": "sg-09dfb82d21e4846c4",
  "snapshot_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)

The key point is the value of snapsho_id.
Specifying an encrypted snapshot ID here will create a DB instance from this.

Execute the following functions

Resources:
  Function4:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          REGION: !Ref AWS::Region
      Code:
        ZipFile: |
          import boto3
          import os
          
          region = os.environ['REGION']
          
          client = boto3.client('rds', region_name=region)
          
          def lambda_handler(event, context):
            instance_id = event['instance_id']
            snapshot_id = event['snapshot_id']
            availability_zone = event['availability_zone']
            db_subnet_group_name = event['db_subnet_group_name']
            security_group_id = event['security_group_id']
            
            response = client.restore_db_instance_from_db_snapshot(
              DBInstanceIdentifier='{instance}-encrypted'.format(instance=instance_id),
              DBSnapshotIdentifier=snapshot_id,
              AvailabilityZone=availability_zone,
              DBSubnetGroupName=db_subnet_group_name,
              VpcSecurityGroupIds=[
                security_group_id,
              ]
            )
            return response['DBInstance']['DBInstanceIdentifier']
      FunctionName: !Sub "${Prefix}-function4"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

Execute the restore_db_instance_from_db_snapshot method to create a DB instance from the snapshot.

The return value is the ID of the DB instance created.

In the state machine, the key named encrypted_instance_id stores the ID of the DB instance.
In response, the data held by the state machine is updated.
The following is an example.

{
  "availability_zone": "ap-northeast-1d",
  "instance_id": "soa-04-005",
  "no_encrypted_snapshot_id": "soa-04-005-no-encrypted",
  "db_subnet_group_name": "dbsubnetgroup",
  "security_group_id": "sg-09dfb82d21e4846c4",
  "no_encrypted_snapshot_status": "creating",
  "encrypted_snapshot_id": "soa-04-005-encrypted",
  "encrypted_instance_id": "soa-04-005-encrypted"
}
Code language: JSON / JSON with Comments (json)

(Reference) RDS

Resources:
  DBInstance:
    Type: AWS::RDS::DBInstance
    DeletionPolicy: Delete
    Properties:
      AllocatedStorage: !Ref DBAllocatedStorage
      AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone}"
      DBInstanceClass: !Ref DBInstanceClass
      DBInstanceIdentifier: !Ref Prefix
      DBSubnetGroupName: !Ref DBSubnetGroup
      Engine: !Ref DBEngine
      EngineVersion: !Ref DBEngineVersion
      MasterUsername: !Ref DBMasterUsername
      MasterUserPassword: !Ref DBMasterUserPassword
      StorageEncrypted: false
      VPCSecurityGroups:
        - !Ref DBSecurityGroup
        
  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupName: dbsubnetgroup
      DBSubnetGroupDescription: testgroup.
      SubnetIds:
        - !Ref DBSubnet1
        - !Ref DBSubnet2
Code language: YAML (yaml)

No special configuration is performed.
Create a DB instance.

(Reference) EC2

Resources:
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      IamInstanceProfile: !Ref InstanceProfile
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      NetworkInterfaces:
        - DeviceIndex: 0
          SubnetId: !Ref InstanceSubnet
          GroupSet:
            - !Ref InstanceSecurityGroup
      UserData: !Base64 |
        #!/bin/bash -xe
        yum update -y
        yum install -y mariadb
Code language: YAML (yaml)

User data defines the initialization process.
For more information on user data, see the following page.

MariaDBをインストールすることで、DB接続用のクライアントを準備します。
詳細につきましては、以下のページをご確認ください。

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormations stack and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • EC2 instance: i-0c948c4d3fa569462
  • DB Instance: soa-04-005
  • DB instance endpoint: soa-04-005.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com
  • Lambda function 1: soa-04-005-function1
  • Lambda function 2: soa-04-005-function2
  • Lambda function 3: soa-04-005-function3
  • Lambda function 4: soa-04-005-function4
  • Step Functions state machine: soa-04-005

Check the DB instance from the AWS Management Console.

Detail of RDS 1.

Storage shows that Encryption is “Not enabled”.
In other words, it is not encrypted.

Check the Step Functions state machine.

Detail of Step Functions 1.

Indeed, a state machine has been created.
The flow of the state machine is represented as a graph.

Operation Check

Connect to unencrypted DB instance

Now that you are ready, access the EC2 instance.

SSM Session Manager is used to access EC2 instances.

% aws ssm start-session --target i-0c948c4d3fa569462
...
sh-4.2$
Code language: Bash (bash)

For more information on SSM Session Manager, please refer to the following page.

Check the installation status of the MySQL client.

sh-4.2$ mysql -V
mysql  Ver 15.1 Distrib 5.5.68-MariaDB, for Linux (aarch64) using readline 5.1
Code language: Bash (bash)

Indeed, the MySQL client is installed by user data.

Use this client package to connect to the DB instance.

sh-4.2$ mysql -h soa-04-005.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com -P 3306 -u testuser -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 8.0.28 Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> 
Code language: Bash (bash)

Connection made.

Create a test database and tables to store test data.

MySQL [(none)]> CREATE database test;
Query OK, 1 row affected (0.01 sec)

MySQL [(none)]> use test;
Database changed

MySQL [test]> CREATE TABLE planet (id INT UNSIGNED AUTO_INCREMENT, name VARCHAR(30), PRIMARY KEY(id));
Query OK, 0 rows affected (0.02 sec)

MySQL [test]> INSERT INTO planet (name) VALUES ("Mercury");
Query OK, 1 row affected (0.00 sec)

MySQL [test]> select * from planet;
+----+---------+
| id | name    |
+----+---------+
|  1 | Mercury |
+----+---------+
1 row in set (0.00 sec)
Code language: Bash (bash)

Test data could be written.

Step Functions state machine execution

Run the state machine.

Set arguments to Input.
The argument is the ID of the unencrypted DB instance.

Detail of Step Functions 2.

Press Start execution.

Detail of Step Functions 3.

State machine execution has started.
Now each state will be executed in turn.

Detail of Step Functions 4.

Paused at WaitState1.
Input shows that the value of no_encypted_snapshot_status is “creating”.
In other words, we are waiting for the status of the snapshot created from the DB instance to change to “available”.

Check the details of this snapshot.

Detail of RDS 2.

Looking at the Status, it is indeed “creating”.
In addition, the KMS key ID is not set, so it is not encrypted.

After a short wait, the process moves to the next state.

Detail of Step Functions 5.

Paused at WaitState2.
If you look at the Input, you will see that the value of encypted_snapshot_status is “creating”.
This snapshot was created by copying the previous snapshot with the encryption option enabled.
We are waiting for the status of this to change to “available”.

Check the details of this snapshot.

Detail of RDS 3.

Looking at the Status, it is indeed “creating”.
In addition, the KMS key ID is set, so it is encrypted.

After waiting for a while, processing of all states is completed.

Detail of Step Functions 6.

A new DB instance has been created from the encrypted snapshot.
The ID of the DB instance is “soa-04-005-encrypted”.

Encrypted DB instance

Check the newly created DB instance.

Detail of RDS 4.

Encryption shows “enabled”.
This indicates that it is indeed encrypted.

Finally, access this DB instance.

sh-4.2$ mysql -h soa-04-005-encrypted.cl50iikpthxs.ap-northeast-1.rds.amazonaws.com -P 3306 -u testuser -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.28 Source distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> 
Code language: Bash (bash)

I was able to log in successfully.

Check the stored data.

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)

MySQL [(none)]> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MySQL [test]> 

MySQL [test]> show tables;
+----------------+
| Tables_in_test |
+----------------+
| planet         |
+----------------+
1 row in set (0.01 sec)

MySQL [test]> select * from planet;
+----+---------+
| id | name    |
+----+---------+
|  1 | Mercury |
+----+---------+
1 row in set (0.00 sec)
Code language: Bash (bash)

Data was saved.
This was saved before encryption.

From the above, we have created an encrypted DB instance with the same contents as the unencrypted DB instance.

Summary

We have shown how to encrypt an unencrypted RDS DB instance using Step Functions.

タイトルとURLをコピーしました