Connect to RDS from Lambda in VPC via RDS Proxy
Consider a configuration where Lambda is deployed in a VPC and connects to RDS.
When accessing RDS from Lambda, it is best practice to connect via RDS Proxy rather than directly.
Many applications, including those built on modern serverless architectures, can have a large number of open connections to the database server, and may open and close database connections at a high rate, exhausting database memory and compute resources. Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency and application scalability.
Amazon RDS Proxy
In this example, we will use the RDS Proxy to verify the configuration of connecting to the RDS from Lambda.
Environment
Create three subnets in the VPC.
One is for Lambda. Normally, Lambda is created outside the VPC, but as shown here, it can also be created inside the VPC.
The other two are for RDS. In this case, one DB instance is created.
The Lambda function will not connect directly to the DB instance endpoint, but to the RDS Proxy endpoint.
Register the DB instance as a target group in the RDS Proxy so that it can connect to the DB instance via the RDS Proxy.
The function will be created in Python 3.8.
Create an API Gateway and set up a Lambda function as the backend.
The idea is to set up the Lambda function so that it is executed by accessing the API Gateway endpoint.
The following is a brief description of the behavior of the configuration we will create.
Create a table to store date/time data in a DB instance.
When an HTTP request arrives at the API Gateway, the back-end Lambda is called to retrieve the current date and time.
The acquired date/time data is stored in the DB instance via RDS Proxy.
In addition, all data stored in the DB instance is retrieved and returned as an HTTP response.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/037
Explanation of key points of the template files
Creating Proxy and Target Group
This page covers the contents related to RDS Proxy.
For RDS itself, please refer to the following page
First, check the RDS Proxy itself.
Resources:
DBProxy:
Type: AWS::RDS::DBProxy
Properties:
Auth:
- IAMAuth: DISABLED
AuthScheme: SECRETS
SecretArn: !Ref Secret
DBProxyName: !Sub "${Prefix}-DBProxy"
EngineFamily: !Ref DBProxyEngineFamily
IdleClientTimeout: 120
RequireTLS: false
RoleArn: !GetAtt DBProxyRole.Arn
VpcSecurityGroupIds:
- !Ref DBProxySecurityGroup
VpcSubnetIds:
- !Ref DBSubnet1
- !Ref DBSubnet2
Code language: YAML (yaml)Set the authentication method for connecting to the RDS (DB instance) in the Auth property.
In this case, we will use the secrets of the Secrets Manager for authentication.
The EngineFamily property specifies the type of DB instance to connect to.
This time, set “MYSQL” since it is a MySQL type DB instance.
Set the security group to be applied to RDS Proxy in the VpcSecurityGroupIds property.
This time we will connect to a MySQL type DB instance, so we will apply a security group that allows inbound 3306/tcp communication from the Lambda subnet.
In the VpcSubnetIds property, specify the subnet to which you want to associate the RDS Proxy.
The target subnet will be the one where the DB instance is located.
The DB instance to which the RDS Proxy will be connected is defined in the target group.
Resources:
DBProxyTargetGroup:
Type: AWS::RDS::DBProxyTargetGroup
Properties:
DBProxyName: !Ref DBProxy
DBInstanceIdentifiers:
- !Ref DBInstance
TargetGroupName: default
ConnectionPoolConfigurationInfo:
MaxConnectionsPercent: 100
MaxIdleConnectionsPercent: 50
ConnectionBorrowTimeout: 120
Code language: YAML (yaml)Specify the DB instance in the DBInstanceIdentifiers property.
Check the Secrets Manager secret.
Resources:
Secret:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub "${Prefix}-Secret"
SecretString: !Sub '{"username":"${DBMasterUsername}","password":"${DBMasterUserPassword}"}'
Code language: YAML (yaml)Specify the username and password set for the DB instance in JSON format to create the secret.
Create an IAM role to be associated with the RDS Proxy.
Resources:
DBProxyRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- rds.amazonaws.com
Policies:
- PolicyName: !Sub "${Prefix}-DBProxyPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: secretsmanager:GetSecretValue
Resource: !Ref Secret
- Effect: Allow
Action: kms:Decrypt
Resource: '*' # use default key.
Code language: YAML (yaml)Grant permission to retrieve the encrypted value from the secret defined earlier and to decrypt the KMS key used for this purpose.
The following page was used to set this up
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-setup.html#rds-proxy-secrets-arns
Lambda function to connect to RDS Proxy
Check the Lambda function.
Resources:
Function:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
DB_ENDPOINT_PORT: !Ref MySQLPort
DB_NAME: !Ref DBName
DB_PASSWORD: !Ref DBMasterUserPassword
DB_PROXY_ENDPOINT_ADDRESS: !Ref DBProxyEndpointAddress
DB_TABLENAME: !Ref DBTableName
DB_USER: !Ref DBMasterUsername
REGION: !Ref AWS::Region
Code:
S3Bucket: !Ref CodeS3Bucket
S3Key: !Ref CodeS3Key
FunctionName: !Sub "${Prefix}-function"
Handler: index.lambda_handler
Runtime: python3.8
Role: !GetAtt FunctionRole.Arn
Timeout: 10
VpcConfig:
SecurityGroupIds:
- !Ref FunctionSecurityGroup
SubnetIds:
- !Ref FunctionSubnet
Code language: YAML (yaml)This time, we will create a Lambda function from a deployment package (Zip file) uploaded to an S3 bucket.
Please refer to the following page for basic information on Lambda functions.
There are two key points.
The first is to set environment variables.
The Environment and Variables properties allow you to define variables that can be used within the function.
In this case, we will define the RDS Proxy endpoint to connect to the RDS Proxy and DB instances, user names and passwords, table names, etc.
The second is to configure settings related to the VPC.
This time, since the Lambda function will be set up inside the VPC, we will specify the subnet where it will be installed and the security group to be applied.
The content of the security group is to allow HTTPS (443/tcp) inbound communication.
Check the script to be executed by the function.
import boto3
import datetime
import json
import mysql.connector
import os
db_name = os.environ['DB_NAME']
db_password = os.environ['DB_PASSWORD']
db_proxy_endpoint_address = os.environ['DB_PROXY_ENDPOINT_ADDRESS']
db_tablename = os.environ['DB_TABLENAME']
db_user = os.environ['DB_USER']
region = os.environ['REGION']
def lambda_handler(event, context):
conn = mysql.connector.connect(
host=db_proxy_endpoint_address,
port=db_endpoint_port,
user=db_user,
password=db_password,
database=db_name
)
cur = conn.cursor()
table_sql = 'create table if not exists {db}.{tbl} (dt datetime);'.format(
db=db_name,
tbl=db_tablename
)
cur.execute(table_sql)
now = datetime.datetime.now()
now_str = now.strftime('%Y-%m-%d %H:%M:%S')
write_sql = 'insert into {tbl} values ("{now}");'.format(
tbl=db_tablename,
now=now_str
)
cur.execute(write_sql)
cur.close()
conn.commit()
cur = conn.cursor()
read_sql = 'select * from {tbl};'.format(tbl=db_tablename)
cur.execute(read_sql)
content = [record[0].strftime('%Y-%m-%d %H:%M:%S') for record in cur]
cur.close()
conn.close()
return {
'statusCode': 200,
'body': json.dumps(content, indent=2)
}
Code language: Python (python)This page will focus on the RDS Proxy, so we will not go into details of the Lambda function itself.
We will review the points for connecting to the RDS Proxy.
Get environment variables from os.environ.
These are the variables set when defining the Lambda function described above.
A particularly important variable is the RDS Proxy endpoint.
The connection is made towards the RDS Proxy endpoint, not the endpoint created in the DB instance.
To connect to MySQL from Python, we will use mysql-connector-python, the official MySQL driver.
When creating the Zip file to deploy, download the package locally and deploy it together using the following command.
pip3 install mysql-connector-python -t .
zip -r deploy.zip .
Code language: Bash (bash)Here is a brief description of the code.
- connect to the DB instance via RDS Proxy by referencing environment variables.
- create a table to store the current date and time for the first time only. 3. retrieve the current date and time.
- retrieve the current date/time and store it in the table. 4.
- retrieve the date/time data stored in the table and return it to the user.
Set up Lambda for API Gateway backend
To create an API Gateway, the following five resources must be created
- API Gateway main body
- Stage
- Integration
- Root
- IAM Roles/Permissions
Please see the following page for details.
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- DB instance: fa-037-dbinstance
- RDS Proxy: fa-037-dbproxy
- API Gateway endpoint: https://oeebpxb1r2.execute-api.ap-northeast-1.amazonaws.com/
The resource creation status is also checked from the AWS Management Console.
Check the RDS Proxy.
RDS Proxy has been successfully created.
An endpoint has been created.
The Lambda function will access this endpoint.
A DB instance is associated with the target group of the RDS Proxy.
When you connect to this RDS Proxy, you are connecting to the associated DB instance.
Authentication is configured using the Secrets Manager.
Confirmation of Operation
Now that everything is ready, access the API Gateway.
The response is returned normally.
Date and time data is displayed more frequently each time it is accessed.
In this way, when accessing a DB instance from Lambda, a connection can be made via RSD Proxy.
Summary
We have confirmed how to connect from Lambda to DB instances via RDS Proxy.