Connect to RDS from Lambda in VPC via RDS Proxy

TOC

Connect to RDS from Lambda in VPC via RDS Proxy

Consider a configuration where Lambda is deployed in a VPC and connects to RDS.
When accessing RDS from Lambda, it is best practice to connect via RDS Proxy rather than directly.

Many applications, including those built on modern serverless architectures, can have a large number of open connections to the database server, and may open and close database connections at a high rate, exhausting database memory and compute resources. Amazon RDS Proxy allows applications to pool and share connections established with the database, improving database efficiency and application scalability.
Amazon RDS Proxy

In this example, we will use the RDS Proxy to verify the configuration of connecting to the RDS from Lambda.

Environment

Diagram of connect from Lambda in VPC via RDS Proxy

Create three subnets in the VPC.
One is for Lambda. Normally, Lambda is created outside the VPC, but as shown here, it can also be created inside the VPC.
The other two are for RDS. In this case, one DB instance is created.

The Lambda function will not connect directly to the DB instance endpoint, but to the RDS Proxy endpoint.
Register the DB instance as a target group in the RDS Proxy so that it can connect to the DB instance via the RDS Proxy.
The function will be created in Python 3.8.

Create an API Gateway and set up a Lambda function as the backend.
The idea is to set up the Lambda function so that it is executed by accessing the API Gateway endpoint.

The following is a brief description of the behavior of the configuration we will create.
Create a table to store date/time data in a DB instance.
When an HTTP request arrives at the API Gateway, the back-end Lambda is called to retrieve the current date and time.
The acquired date/time data is stored in the DB instance via RDS Proxy.
In addition, all data stored in the DB instance is retrieved and returned as an HTTP response.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL

https://github.com/awstut-an-r/awstut-fa/tree/main/037

Explanation of key points of the template files

Creating Proxy and Target Group

This page covers the contents related to RDS Proxy.
For RDS itself, please refer to the following page

First, check the RDS Proxy itself.

Resources:
  DBProxy:
    Type: AWS::RDS::DBProxy
    Properties:
      Auth:
        - IAMAuth: DISABLED
          AuthScheme: SECRETS
          SecretArn: !Ref Secret
      DBProxyName: !Sub "${Prefix}-DBProxy"
      EngineFamily: !Ref DBProxyEngineFamily
      IdleClientTimeout: 120
      RequireTLS: false
      RoleArn: !GetAtt DBProxyRole.Arn
      VpcSecurityGroupIds:
        - !Ref DBProxySecurityGroup
      VpcSubnetIds:
        - !Ref DBSubnet1
        - !Ref DBSubnet2
Code language: YAML (yaml)

Set the authentication method for connecting to the RDS (DB instance) in the Auth property.
In this case, we will use the secrets of the Secrets Manager for authentication.

The EngineFamily property specifies the type of DB instance to connect to.
This time, set “MYSQL” since it is a MySQL type DB instance.

Set the security group to be applied to RDS Proxy in the VpcSecurityGroupIds property.
This time we will connect to a MySQL type DB instance, so we will apply a security group that allows inbound 3306/tcp communication from the Lambda subnet.

In the VpcSubnetIds property, specify the subnet to which you want to associate the RDS Proxy.
The target subnet will be the one where the DB instance is located.

The DB instance to which the RDS Proxy will be connected is defined in the target group.

Resources:
  DBProxyTargetGroup:
    Type: AWS::RDS::DBProxyTargetGroup
    Properties:
      DBProxyName: !Ref DBProxy
      DBInstanceIdentifiers:
        - !Ref DBInstance
      TargetGroupName: default
      ConnectionPoolConfigurationInfo:
        MaxConnectionsPercent: 100
        MaxIdleConnectionsPercent: 50
        ConnectionBorrowTimeout: 120
Code language: YAML (yaml)

Specify the DB instance in the DBInstanceIdentifiers property.

Check the Secrets Manager secret.

Resources:
  Secret:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub "${Prefix}-Secret"
      SecretString: !Sub '{"username":"${DBMasterUsername}","password":"${DBMasterUserPassword}"}'
Code language: YAML (yaml)

Specify the username and password set for the DB instance in JSON format to create the secret.

Create an IAM role to be associated with the RDS Proxy.

Resources:
  DBProxyRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - rds.amazonaws.com
      Policies:
        - PolicyName: !Sub "${Prefix}-DBProxyPolicy"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action: secretsmanager:GetSecretValue
                Resource: !Ref Secret
              - Effect: Allow
                Action: kms:Decrypt
                Resource: '*' # use default key.
Code language: YAML (yaml)

Grant permission to retrieve the encrypted value from the secret defined earlier and to decrypt the KMS key used for this purpose.
The following page was used to set this up

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-setup.html#rds-proxy-secrets-arns

Lambda function to connect to RDS Proxy

Check the Lambda function.

Resources:
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          DB_ENDPOINT_PORT: !Ref MySQLPort
          DB_NAME: !Ref DBName
          DB_PASSWORD: !Ref DBMasterUserPassword
          DB_PROXY_ENDPOINT_ADDRESS: !Ref DBProxyEndpointAddress
          DB_TABLENAME: !Ref DBTableName
          DB_USER: !Ref DBMasterUsername
          REGION: !Ref AWS::Region
      Code:
        S3Bucket: !Ref CodeS3Bucket
        S3Key: !Ref CodeS3Key
      FunctionName: !Sub "${Prefix}-function"
      Handler: index.lambda_handler
      Runtime: python3.8
      Role: !GetAtt FunctionRole.Arn
      Timeout: 10
      VpcConfig:
        SecurityGroupIds:
          - !Ref FunctionSecurityGroup
        SubnetIds:
          - !Ref FunctionSubnet
Code language: YAML (yaml)

This time, we will create a Lambda function from a deployment package (Zip file) uploaded to an S3 bucket.
Please refer to the following page for basic information on Lambda functions.

There are two key points.

The first is to set environment variables.
The Environment and Variables properties allow you to define variables that can be used within the function.
In this case, we will define the RDS Proxy endpoint to connect to the RDS Proxy and DB instances, user names and passwords, table names, etc.

The second is to configure settings related to the VPC.
This time, since the Lambda function will be set up inside the VPC, we will specify the subnet where it will be installed and the security group to be applied.
The content of the security group is to allow HTTPS (443/tcp) inbound communication.

Check the script to be executed by the function.

import boto3
import datetime
import json
import mysql.connector
import os


db_name = os.environ['DB_NAME']
db_password = os.environ['DB_PASSWORD']
db_proxy_endpoint_address = os.environ['DB_PROXY_ENDPOINT_ADDRESS']
db_tablename = os.environ['DB_TABLENAME']
db_user = os.environ['DB_USER']
region = os.environ['REGION']


def lambda_handler(event, context):
    conn = mysql.connector.connect(
        host=db_proxy_endpoint_address,
        port=db_endpoint_port,
        user=db_user,
        password=db_password,
        database=db_name
        )
    cur = conn.cursor()

    table_sql = 'create table if not exists {db}.{tbl} (dt datetime);'.format(
        db=db_name,
        tbl=db_tablename
        )
    cur.execute(table_sql)

    now = datetime.datetime.now()
    now_str = now.strftime('%Y-%m-%d %H:%M:%S')
    write_sql = 'insert into {tbl} values ("{now}");'.format(
        tbl=db_tablename,
        now=now_str
    )
    cur.execute(write_sql)

    cur.close()
    conn.commit()

    cur = conn.cursor()
    read_sql = 'select * from {tbl};'.format(tbl=db_tablename)
    cur.execute(read_sql)
    content = [record[0].strftime('%Y-%m-%d %H:%M:%S') for record in cur]

    cur.close()
    conn.close()


    return {
        'statusCode': 200,
        'body': json.dumps(content, indent=2)
    }
Code language: Python (python)

This page will focus on the RDS Proxy, so we will not go into details of the Lambda function itself.
We will review the points for connecting to the RDS Proxy.

Get environment variables from os.environ.
These are the variables set when defining the Lambda function described above.
A particularly important variable is the RDS Proxy endpoint.
The connection is made towards the RDS Proxy endpoint, not the endpoint created in the DB instance.

To connect to MySQL from Python, we will use mysql-connector-python, the official MySQL driver.
When creating the Zip file to deploy, download the package locally and deploy it together using the following command.

pip3 install mysql-connector-python -t .

zip -r deploy.zip .
Code language: Bash (bash)

Here is a brief description of the code.

connect to the DB instance via RDS Proxy by referencing environment variables.
create a table to store the current date and time for the first time only. 3. retrieve the current date and time.
retrieve the current date/time and store it in the table. 4.
retrieve the date/time data stored in the table and return it to the user.

Set up Lambda for API Gateway backend

To create an API Gateway, the following five resources must be created

API Gateway main body
Stage
Integration
Root
IAM Roles/Permissions

Please see the following page for details.

Architecting

Use CloudFormation to build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

DB instance: fa-037-dbinstance
RDS Proxy: fa-037-dbproxy
API Gateway endpoint: https://oeebpxb1r2.execute-api.ap-northeast-1.amazonaws.com/

The resource creation status is also checked from the AWS Management Console.
Check the RDS Proxy.

RDS Proxy has been successfully created.

An endpoint has been created.
The Lambda function will access this endpoint.

A DB instance is associated with the target group of the RDS Proxy.
When you connect to this RDS Proxy, you are connecting to the associated DB instance.

Authentication is configured using the Secrets Manager.

Confirmation of Operation

Now that everything is ready, access the API Gateway.

The response is returned normally.
Date and time data is displayed more frequently each time it is accessed.
In this way, when accessing a DB instance from Lambda, a connection can be made via RSD Proxy.

Summary

We have confirmed how to connect from Lambda to DB instances via RDS Proxy.