DVA_EN

Introduction to SQS FIFO Queues

スポンサーリンク
Introduction to SQS FIFO Queues. DVA_EN
スポンサーリンク
スポンサーリンク

Introduction to SQS FIFO Queues

SQS has two types of queues, and we will touch on the FIFO queue.

FIFO (First-In-First-Out) queues have all the capabilities of the standard queues, but are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated.

Amazon SQS FIFO (First-In-First-Out) queues

This page creates a FIFO queue and a standard queue and compares their settings and behavior.

Environment

Diagram of SQS FIFO queue vs standard queue.

This configuration places SQS between two Lambda functions.

The SQS queue will have a FIFO queue and a standard queue.

Lambda function 1 sends messages to the two queues.
Lambda function 2 polls both queues and receives and processes any messages in the queues.
The runtime environment for the Lambda function is Python 3.8.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL

https://github.com/awstut-an-r/awstut-dva/tree/main/03/008

Explanation of key points of template files

This page focuses on the comparison of FIFO queues and standard queues.

For information on how to trigger a Lambda function on an SQS queue, please see the following page.

SQS

First, check the standard queue.

Resources:
  StandardQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${Prefix}-standard"
      ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
      VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)

No special settings are made.
Set only a 20 second long poll (RecieveMessageWaitTimeSeconds) and a 90 second visibility timeout (VisibilityTimeout).

Next, check the FIFO queue.

Resources:      
  FifoQueue:
    Type: AWS::SQS::Queue
    Properties:
      ContentBasedDeduplication: true
      FifoQueue: true
      QueueName: !Sub "${Prefix}.fifo"
      ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
      VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)

There are three points.

The first is the cue name.
The naming convention is as follows.

The name of a FIFO queue can only include alphanumeric characters, hyphens, or underscores, must end with .fifo suffix and be 1 to 80 in length.

Set the QueueName property according to the above.

The second is a duplicate exclusion setting.
FIFO queues do not allow duplicates.

Unlike standard queues, FIFO queues don’t introduce duplicate messages. FIFO queues help you avoid sending duplicates to a queue. If you retry the SendMessage action within the 5-minute deduplication interval, Amazon SQS doesn’t introduce any duplicates into the queue.

Exactly-once processing

There are two ways to set up deduplication, but the easiest way is to activate content-based exclusion.

Enable content-based deduplication. This instructs Amazon SQS to use a SHA-256 hash to generate the message deduplication ID using the body of the message—but not the attributes of the message.

Exactly-once processing

Content-based deduplication can be enabled by setting the ContentBasedDeduplication property to true.

The third is the FifoQueue property.
Setting true to this property will construct this queue as a FIFO queue.

Lambda Functions

Function 1

Resources:
  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import boto3
          import datetime
          import os
          
          n = 10
          
          fifo_queue_url = os.environ['FIFO_QUEUE_URL']
          message_group_id = os.environ['MESSAGE_GROUP_ID']
          standard_queue_url = os.environ['STANDARD_QUEUE_URL']
          region_name = os.environ['REGION_NAME']
            
          client = boto3.client('sqs', region_name=region_name)
          
          def lambda_handler(event, context):
            for i in range(n):
              now = datetime.datetime.now()
              now_str = now.strftime('%Y-%m-%d %H:%M:%S:%f')
              body = '{n}: {datetime}'.format(n=i, datetime=now_str)
              
              response_stndard = client.send_message(
                QueueUrl=standard_queue_url,
                MessageBody=body
              )
              print(response_stndard)
              
              response_fifo = client.send_message(
                QueueUrl=fifo_queue_url,
                MessageBody=body,
                MessageGroupId=message_group_id
              )
              print(response_fifo)
      Environment:
        Variables:
          FIFO_QUEUE_URL: !Ref FifoQueueUrl
          MESSAGE_GROUP_ID: !Ref Prefix
          REGION_NAME: !Ref AWS::Region
          STANDARD_QUEUE_URL: !Ref StandardQueueUrl
      FunctionName: !Sub "${Prefix}-Function1"
      Handler: !Ref LambdaHandler
      MemorySize: !Ref LambdaMemorySize
      Runtime: !Ref LambdaRuntime
      Role: !GetAtt LambdaRole1.Arn
      Timeout: !Ref LambdaTimeout
Code language: YAML (yaml)

The code to be executed by the Lambda function in inline format.
For more information, please see the following page.

Create a client object for boto3 SQS.
To send a message to the SQS queue, use the send_message method of the same object.

This function calls the same method twice.
The first call is for the standard queue and the second call is for the FIFO queue.

Check from sending a message to a standard queue.
To send a message, specify the URL and body of the queue.

In contrast, to send a message to a FIFO queue, specify a message group ID in addition to the above two parameters.

FIFO queue logic applies only per message group ID. Each message group ID represents a distinct ordered message group within an Amazon SQS queue. For each message group ID, all messages are sent and received in strict order.

FIFO delivery logic

This time, a string (dva-03-008) common to all messages is set as the message group ID.
This means that all messages sent to this queue will be strictly ordered.

The following is the IAM role for this function.

Resources:
  LambdaRole1:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: SendSQSMessagePolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sqs:GetQueueUrl
                  - sqs:SendMessage
                Resource:
                  - !Ref FifoQueueArn
                  - !Ref StandardQueueArn
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)

Authorization to send messages to two SQS queues.

(Reference) Function 2

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          def lambda_handler(event, context):
            print(event)
      FunctionName: !Sub "${Prefix}-Function2"
      Handler: !Ref LambdaHandler
      MemorySize: !Ref LambdaMemorySize
      Runtime: !Ref LambdaRuntime
      Role: !GetAtt LambdaRole2.Arn
      Timeout: !Ref LambdaTimeout
Code language: YAML (yaml)

A Lambda function triggered by SQS.
This is a simple content that outputs an event object.

The following is the IAM role for this function.

Resources:
  LambdaRole2:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      Policies:
        - PolicyName: GetSQSMEssagePolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sqs:ReceiveMessage
                  - sqs:DeleteMessage
                  - sqs:GetQueueAttributes
                Resource:
                  - !Ref StandardQueueArn
                  - !Ref FifoQueueArn
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)

Authorization to retrieve messages from two SQS queues.

Create event source mappings to associate SQS queues with Lambda functions.

Resources:
  EventSourceMapping1:
    Type: AWS::Lambda::EventSourceMapping
    Properties: 
      BatchSize: !Ref BatchSize
      Enabled: true
      EventSourceArn: !Ref StandardQueueArn
      FunctionName: !Ref Function2
      
  EventSourceMapping2:
    Type: AWS::Lambda::EventSourceMapping
    Properties: 
      BatchSize: !Ref BatchSize
      Enabled: true
      EventSourceArn: !Ref FifoQueueArn
      FunctionName: !Ref Function2
Code language: YAML (yaml)

Associate two SQS queues with one function.

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stack

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • SQS Standard Queue: dva-03-008-standard
  • SQS FIFO queue: dva-03-008.fifo
  • Lambda Function 1: dva-03-008-Function1
  • Lambda Function 2: dva-03-008-Function2

Check each resource from the AWS Management Console.

Check the SQS queue.

Detail of SQS 1.
Detail of SQS 2.

Indeed, two SQS queues are created.

One is standard and the other is FIFO queue.

Lambda function 2 (described below) is set as the Lambda trigger.

Check the Lambda function.

Detail of Lambda 1.
Detail of Lambda 2.

Two functions have been successfully created.

Operation Check

Now that we are ready, we execute function 1.

The following are the fruits of the execution.

Detail of Lambda 3.

Completed successfully.

Next, check the execution log of Lambda function 2.

Detail of Lambda 4.

The above is a portion of the log, which indeed shows that messages are received from the standard and FIFO queues.

This means that this function was triggered from two SQS queues.

For reference, check the order in which this function retrieves messages from the SQS queue.
The body of the message consists of “[number]: 2024/03/02“, where the numbers 0~9 are taken in order.
That is, by looking at this number, you can determine if you have received the message in order.

We tried this verification 10 times.
As a result, the standard queue failed to receive messages in order twice.
In contrast, the FIFO queue received all messages in order.

For requirements that require messages to be processed in sequence, consider using a FIFO queue.

Summary

We created a FIFO queue and a standard queue and compared their settings and behavior.

タイトルとURLをコピーしました