Introduction to SQS FIFO Queues
SQS has two types of queues, and we will touch on the FIFO queue.
FIFO (First-In-First-Out) queues have all the capabilities of the standard queues, but are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated.
Amazon SQS FIFO (First-In-First-Out) queues
This page creates a FIFO queue and a standard queue and compares their settings and behavior.
Environment
This configuration places SQS between two Lambda functions.
The SQS queue will have a FIFO queue and a standard queue.
Lambda function 1 sends messages to the two queues.
Lambda function 2 polls both queues and receives and processes any messages in the queues.
The runtime environment for the Lambda function is Python 3.8.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-dva/tree/main/03/008
Explanation of key points of template files
This page focuses on the comparison of FIFO queues and standard queues.
For information on how to trigger a Lambda function on an SQS queue, please see the following page.
SQS
First, check the standard queue.
Resources:
StandardQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub "${Prefix}-standard"
ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)No special settings are made.
Set only a 20 second long poll (RecieveMessageWaitTimeSeconds) and a 90 second visibility timeout (VisibilityTimeout).
Next, check the FIFO queue.
Resources:
FifoQueue:
Type: AWS::SQS::Queue
Properties:
ContentBasedDeduplication: true
FifoQueue: true
QueueName: !Sub "${Prefix}.fifo"
ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)There are three points.
The first is the cue name.
The naming convention is as follows.
The name of a FIFO queue can only include alphanumeric characters, hyphens, or underscores, must end with .fifo suffix and be 1 to 80 in length.
Set the QueueName property according to the above.
The second is a duplicate exclusion setting.
FIFO queues do not allow duplicates.
Unlike standard queues, FIFO queues don’t introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn’t introduce any duplicates into the queue.
Exactly-once processing
There are two ways to set up deduplication, but the easiest way is to activate content-based exclusion.
Enable content-based deduplication. This instructs Amazon SQS to use a SHA-256 hash to generate the message deduplication ID using the body of the message—but not the attributes of the message.
Exactly-once processing
Content-based deduplication can be enabled by setting the ContentBasedDeduplication property to true.
The third is the FifoQueue property.
Setting true to this property will construct this queue as a FIFO queue.
Lambda Functions
Function 1
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
import boto3
import datetime
import os
n = 10
fifo_queue_url = os.environ['FIFO_QUEUE_URL']
message_group_id = os.environ['MESSAGE_GROUP_ID']
standard_queue_url = os.environ['STANDARD_QUEUE_URL']
region_name = os.environ['REGION_NAME']
client = boto3.client('sqs', region_name=region_name)
def lambda_handler(event, context):
for i in range(n):
now = datetime.datetime.now()
now_str = now.strftime('%Y-%m-%d %H:%M:%S:%f')
body = '{n}: {datetime}'.format(n=i, datetime=now_str)
response_stndard = client.send_message(
QueueUrl=standard_queue_url,
MessageBody=body
)
print(response_stndard)
response_fifo = client.send_message(
QueueUrl=fifo_queue_url,
MessageBody=body,
MessageGroupId=message_group_id
)
print(response_fifo)
Environment:
Variables:
FIFO_QUEUE_URL: !Ref FifoQueueUrl
MESSAGE_GROUP_ID: !Ref Prefix
REGION_NAME: !Ref AWS::Region
STANDARD_QUEUE_URL: !Ref StandardQueueUrl
FunctionName: !Sub "${Prefix}-Function1"
Handler: !Ref LambdaHandler
MemorySize: !Ref LambdaMemorySize
Runtime: !Ref LambdaRuntime
Role: !GetAtt LambdaRole1.Arn
Timeout: !Ref LambdaTimeout
Code language: YAML (yaml)The code to be executed by the Lambda function in inline format.
For more information, please see the following page.
Create a client object for boto3 SQS.
To send a message to the SQS queue, use the send_message method of the same object.
This function calls the same method twice.
The first call is for the standard queue and the second call is for the FIFO queue.
Check from sending a message to a standard queue.
To send a message, specify the URL and body of the queue.
In contrast, to send a message to a FIFO queue, specify a message group ID in addition to the above two parameters.
FIFO queue logic applies only per message group ID. Each message group ID represents a distinct ordered message group within an Amazon SQS queue. For each message group ID, all messages are sent and received in strict order.
FIFO delivery logic
This time, a string (dva-03-008) common to all messages is set as the message group ID.
This means that all messages sent to this queue will be strictly ordered.
The following is the IAM role for this function.
Resources:
LambdaRole1:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Policies:
- PolicyName: SendSQSMessagePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sqs:GetQueueUrl
- sqs:SendMessage
Resource:
- !Ref FifoQueueArn
- !Ref StandardQueueArn
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)Authorization to send messages to two SQS queues.
(Reference) Function 2
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
def lambda_handler(event, context):
print(event)
FunctionName: !Sub "${Prefix}-Function2"
Handler: !Ref LambdaHandler
MemorySize: !Ref LambdaMemorySize
Runtime: !Ref LambdaRuntime
Role: !GetAtt LambdaRole2.Arn
Timeout: !Ref LambdaTimeout
Code language: YAML (yaml)A Lambda function triggered by SQS.
This is a simple content that outputs an event object.
The following is the IAM role for this function.
Resources:
LambdaRole2:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
Policies:
- PolicyName: GetSQSMEssagePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sqs:ReceiveMessage
- sqs:DeleteMessage
- sqs:GetQueueAttributes
Resource:
- !Ref StandardQueueArn
- !Ref FifoQueueArn
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)Authorization to retrieve messages from two SQS queues.
Create event source mappings to associate SQS queues with Lambda functions.
Resources:
EventSourceMapping1:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: !Ref BatchSize
Enabled: true
EventSourceArn: !Ref StandardQueueArn
FunctionName: !Ref Function2
EventSourceMapping2:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: !Ref BatchSize
Enabled: true
EventSourceArn: !Ref FifoQueueArn
FunctionName: !Ref Function2
Code language: YAML (yaml)Associate two SQS queues with one function.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stack
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- SQS Standard Queue: dva-03-008-standard
- SQS FIFO queue: dva-03-008.fifo
- Lambda Function 1: dva-03-008-Function1
- Lambda Function 2: dva-03-008-Function2
Check each resource from the AWS Management Console.
Check the SQS queue.
Indeed, two SQS queues are created.
One is standard and the other is FIFO queue.
Lambda function 2 (described below) is set as the Lambda trigger.
Check the Lambda function.
Two functions have been successfully created.
Operation Check
Now that we are ready, we execute function 1.
The following are the fruits of the execution.
Completed successfully.
Next, check the execution log of Lambda function 2.
The above is a portion of the log, which indeed shows that messages are received from the standard and FIFO queues.
This means that this function was triggered from two SQS queues.
For reference, check the order in which this function retrieves messages from the SQS queue.
The body of the message consists of “[number]: [date/time]”, where the numbers 0~9 are taken in order.
That is, by looking at this number, you can determine if you have received the message in order.
We tried this verification 10 times.
As a result, the standard queue failed to receive messages in order twice.
In contrast, the FIFO queue received all messages in order.
For requirements that require messages to be processed in sequence, consider using a FIFO queue.
Summary
We created a FIFO queue and a standard queue and compared their settings and behavior.