Connecting to RDS inside and outside of Lambda function handler methods to measure performance
This section is about refactoring, which is the scope of the AWS DBA.
One of the best practices for connecting to RDS from a Lambda function is to connect to RDS outside of the handler method.
Objects declared outside of the function’s handler method remain initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations.
AWS Lambda execution environment
In this page, we will create two Lambda functions and measure how much performance can be improved by connecting to RDS inside and outside the handler methods, respectively.
Environment
Four subnets will be created in the VPC.
Two are for Lambda.
The other two are for RDS. In this case, we will create one MySQL type DB instance.
Create an RDS Proxy.
By registering a DB instance to the target group of the RDS Proxy, the Lambda function can connect to the DB instance via the RDS Proxy.
Create two Lambda functions.
After connecting to the DB instance, both functions will store and return the current date and time.
The difference between the two functions is the implementation of the part that connects to the DB instance.
Function 1 connects to the DB instance inside the handler method, while Function 2 connects externally.
The runtime environment for the functions is Python 3.8, and Function URLs are created.
Apache Bench is used for verification.
For each Function URL, send 100 requests and measure the time taken to process each request.
CloudFormation Template Files
The above configuration is built using CloudFormation.
The CloudFormation template is located at the following URL
https://github.com/awstut-an-r/awstut-dva/tree/main/04/001
Explanation of key points of template files
This page focuses on the content related to how to connect to a DB instance in a Lambda function.
For information on how to connect to RDS from a Lambda function via RDS Proxy, please refer to the following page
Use mysql-connector-python to connect to a MySQL type DB instance.
Create a Lambda layer and place this package in it.
For more information on the Lambda layer, see the following page
The created Lambda function is executed using a Function URL.
For more information on Function URLs, please refer to the following page
Connecting to DB instance inside handler method
Check the code of the function (function 1) that connects inside the handler method.
import datetime
import mysql.connector
import os
db_endpoint_port = os.environ['DB_ENDPOINT_PORT']
db_name = os.environ['DB_NAME']
db_password = os.environ['DB_PASSWORD']
db_proxy_endpoint_address = os.environ['DB_PROXY_ENDPOINT_ADDRESS']
db_tablename = os.environ['DB_TABLENAME']
db_user = os.environ['DB_USER']
region = os.environ['REGION']
def lambda_handler(event, context):
conn = mysql.connector.connect(
host=db_proxy_endpoint_address,
port=db_endpoint_port,
user=db_user,
password=db_password,
database=db_name
)
cur = conn.cursor()
now = datetime.datetime.now()
now_str = now.strftime('%Y-%m-%d %H:%M:%S')
write_sql = 'insert into {tbl} values ("{now}");'.format(
tbl=db_tablename,
now=now_str
)
cur.execute(write_sql)
cur.close()
conn.commit()
return {
'statusCode': 200,
'body': str(now)
}
Code language: Python (python)Create an object for connection to the DB instance with mysql.connector.connect.
As you can see, it is executed within the handler method.
Connecting to DB instance outside handler method
The following is the code for the function that connects outside the handler method (function 2).
import datetime
import mysql.connector
import os
db_endpoint_port = os.environ['DB_ENDPOINT_PORT']
db_name = os.environ['DB_NAME']
db_password = os.environ['DB_PASSWORD']
db_proxy_endpoint_address = os.environ['DB_PROXY_ENDPOINT_ADDRESS']
db_tablename = os.environ['DB_TABLENAME']
db_user = os.environ['DB_USER']
region = os.environ['REGION']
conn = mysql.connector.connect(
host=db_proxy_endpoint_address,
port=db_endpoint_port,
user=db_user,
password=db_password,
database=db_name
)
def lambda_handler(event, context):
cur = conn.cursor()
now = datetime.datetime.now()
now_str = now.strftime('%Y-%m-%d %H:%M:%S')
write_sql = 'insert into {tbl} values ("{now}");'.format(
tbl=db_tablename,
now=now_str
)
cur.execute(write_sql)
cur.close()
conn.commit()
return {
'statusCode': 200,
'body': str(now)
}
Code language: Python (python)The code is almost identical to the previous one.
The only difference is that the scope in which the connection object is created in mysql.connector.connect is the global scope. This means that the object is defined as a global variable.
(Reference) Initialization process of DB instance
In order to write the current date and time, it is necessary to create a table for date and time data in the DB instance.
This time, a function for initialization processing (Function 3) is prepared and executed.
The code of Function 3 is shown below for reference.
import datetime
import json
import mysql.connector
import os
db_endpoint_port = os.environ['DB_ENDPOINT_PORT']
db_name = os.environ['DB_NAME']
db_password = os.environ['DB_PASSWORD']
db_proxy_endpoint_address = os.environ['DB_PROXY_ENDPOINT_ADDRESS']
db_tablename = os.environ['DB_TABLENAME']
db_user = os.environ['DB_USER']
region = os.environ['REGION']
def lambda_handler(event, context):
conn = mysql.connector.connect(
host=db_proxy_endpoint_address,
port=db_endpoint_port,
user=db_user,
password=db_password,
database=db_name
)
cur = conn.cursor()
table_sql = 'create table if not exists {db}.{tbl} (dt datetime);'.format(
db=db_name,
tbl=db_tablename
)
cur.execute(table_sql)
now = datetime.datetime.now()
now_str = now.strftime('%Y-%m-%d %H:%M:%S')
write_sql = 'insert into {tbl} values ("{now}");'.format(
tbl=db_tablename,
now=now_str
)
cur.execute(write_sql)
cur.close()
conn.commit()
cur = conn.cursor()
read_sql = 'select * from {tbl};'.format(tbl=db_tablename)
cur.execute(read_sql)
content = [record[0].strftime('%Y-%m-%d %H:%M:%S') for record in cur]
cur.close()
conn.close()
return {
'statusCode': 200,
'body': json.dumps(content, indent=2)
}
Code language: Python (python)If the table does not exist, the content is to create a new table.
It then works to retrieve and return all stored data.
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Prepare deployment package for Lambda function
There are three ways to create a Lambda function.
In this case, we will choose the method of uploading a deployment package to an S3 bucket.
For more information, please refer to the following page
Create a CloudFormation stacks and check resources in stacks
Create a CloudFormation stack.
For more information on how to create stacks and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- Function URL for Function 1: https://42lzbfe7dehb5jcnphzp555hly0evucx.lambda-url.ap-northeast-1.on.aws/
- Function URL for function 2: https://6oy3eianevtfz7xw3dopmbnnrm0ijdyr.lambda-url.ap-northeast-1.on.aws/
- Function URL for function 3: https://7poj6cjntirj24de2hlbz2xeia0katuf.lambda-url.ap-northeast-1.on.aws/
Confirmation of Operation
Preliminary Preparation
First, create a table in the DB instance to store date/time data.
Access the Function URL of Function 3 and execute the function.
The table is created and all currently stored data is returned.
The initialization process is successfully completed.
Verification 1: Connecting to DB instance inside handler method
Now that we are ready, we can start the verification.
At a glance, we check the operation of function 1.
The current date and time are returned.
The function is working normally.
We evaluate the performance of this function using Apache Bench.
The following is the execution result.
% ab -n 100 https://42lzbfe7dehb5jcnphzp555hly0evucx.lambda-url.ap-northeast-1.on.aws/
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 42lzbfe7dehb5jcnphzp555hly0evucx.lambda-url.ap-northeast-1.on.aws (be patient).....done
Server Software:
Server Hostname: 42lzbfe7dehb5jcnphzp555hly0evucx.lambda-url.ap-northeast-1.on.aws
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: 42lzbfe7dehb5jcnphzp555hly0evucx.lambda-url.ap-northeast-1.on.aws
Document Path: /
Document Length: 26 bytes
Concurrency Level: 1
Time taken for tests: 50.102 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 27800 bytes
HTML transferred: 2600 bytes
Requests per second: 2.00 [#/sec] (mean)
Time per request: 501.024 [ms] (mean)
Time per request: 501.024 [ms] (mean, across all concurrent requests)
Transfer rate: 0.54 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 50 57 4.5 56 78
Processing: 377 443 96.2 426 1307
Waiting: 377 443 96.2 426 1307
Total: 427 501 96.0 482 1359
Percentage of the requests served within a certain time (ms)
50% 482
66% 507
75% 512
80% 512
90% 534
95% 580
98% 717
99% 1359
100% 1359 (longest request)Code language: plaintext (plaintext)The average processing speed was 501 ms.
Verification 2: Connecting to DB instance outside handler method
The same method is used for verification.
First, check the behavior of function 2.
The current date and time are returned.
The function is working normally.
We evaluate the performance of this function using Apache Bench.
The following is the execution result.
% ab -n 100 https://6oy3eianevtfz7xw3dopmbnnrm0ijdyr.lambda-url.ap-northeast-1.on.aws/
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 6oy3eianevtfz7xw3dopmbnnrm0ijdyr.lambda-url.ap-northeast-1.on.aws (be patient).....done
Server Software:
Server Hostname: 6oy3eianevtfz7xw3dopmbnnrm0ijdyr.lambda-url.ap-northeast-1.on.aws
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128
Server Temp Key: ECDH P-256 256 bits
TLS Server Name: 6oy3eianevtfz7xw3dopmbnnrm0ijdyr.lambda-url.ap-northeast-1.on.aws
Document Path: /
Document Length: 26 bytes
Concurrency Level: 1
Time taken for tests: 11.825 seconds
Complete requests: 100
Failed requests: 0
Total transferred: 27800 bytes
HTML transferred: 2600 bytes
Requests per second: 8.46 [#/sec] (mean)
Time per request: 118.255 [ms] (mean)
Time per request: 118.255 [ms] (mean, across all concurrent requests)
Transfer rate: 2.30 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 47 57 5.1 56 82
Processing: 41 61 75.4 51 798
Waiting: 41 61 75.4 51 798
Total: 90 118 76.2 109 860
Percentage of the requests served within a certain time (ms)
50% 109
66% 113
75% 115
80% 118
90% 124
95% 137
98% 180
99% 860
100% 860 (longest request)Code language: plaintext (plaintext)The average processing speed was 118 ms.
Summary of Verification
The following is a summary of the results of this verification.
All indicators show that the outside connection is faster.
The average value shows that the outside connection was 4.2 times faster than the inside connection in this verification.
Summary
We have verified the extent to which performance varies depending on whether the Lambda function connects to the RDS inside or outside the handler method.
In this verification, we found that connecting outside the handler method is about 4.2 times faster than connecting inside.