AWS_EN

4 destinations for S3 event notifications – SNS/SQS/Lambda/EventBridge

スポンサーリンク
4 destinations for S3 event notifications - SNS/SQS/Lambda/EventBridge AWS_EN
スポンサーリンク
スポンサーリンク

4 destinations for S3 event notifications – SNS/SQS/Lambda/EventBridge

In the following page, we introduced a configuration that automatically creates thumbnail images from uploaded images using the S3 event notification function.

The above page was configured to trigger a Lambda function with an event notification.

This time we will review the four destinations for event notifications.

Environment

Diagram of 4 destinations for S3 event notifications - SNS/SQS/Lambda/EventBridge

Create S3 buckets.
Enable the event notification features.
Specify the following four resources as destinations for notifications.

  • Lambda Function
  • SQS
  • SNS
  • EventBridge

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL.

https://github.com/awstut-an-r/awstut-fa/tree/main/113

Explanation of key points of template files

S3 bucket

Resources:
  Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      BucketName: !Ref BucketName
      NotificationConfiguration:
        LambdaConfigurations: 
          - Event: "s3:ObjectCreated:*"
            Function: !Ref Function1Arn
        QueueConfigurations: 
          - Event: "s3:ObjectTagging:*"
            Queue: !Ref QueueArn
        TopicConfigurations: 
          - Event: "s3:ObjectRemoved:*"
            Topic: !Ref TopicArn
        EventBridgeConfiguration: 
          EventBridgeEnabled: true
Code language: YAML (yaml)

Specify four notification destinations.

The LambdaConfigurations property is used to notify the Lambda function.
The condition for event notification is set to “s3:ObjectCreated:*” and the function will be triggered when an object is placed in the bucket.

The QueueConfigurations property is used to notify SQS.
As for the conditions for event notification, set “s3:ObjectTagging:*” to send a message to the SQS queue when an object in the bucket is tagged (added, etc.).

Notification to SNS is done using the TopicConfigurations property.
The condition for event notification is “s3:ObjectRemoved:*”, which sends a message to the SNS topic when an object is deleted from the bucket.

Notifications to EventBridge use the EventBridgeConfiguration property.
This property enables notification to EventBridge.

Lambda Functions

Resources:
  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Code:
        ZipFile: |
          def lambda_handler(event, context):
            print(event)
      FunctionName: !Sub "${Prefix}-Function1"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt Function1Role.Arn
Code language: YAML (yaml)

Lambda function for event notification destination.

The code to be executed by the Lambda function in inline format.
For more information, please refer to the following page.

The code to be executed is a simple one that outputs the contents of the event object.

Incidentally, SQS, SNS, and EventBridge, described below, also execute functions with similar contents.

Check the permissions when the notification is to a Lambda function.
Triggering a Lambda function in an S3 event notification means that the S3 bucket invokes the Lambda function.
This is set by the Lambda resource-based policy.

Resources:
  S3Permission: 
    Type: AWS::Lambda::Permission
    Properties: 
      FunctionName: !Ref Function1
      Action: lambda:InvokeFunction
      Principal: s3.amazonaws.com
      SourceArn: !Sub "arn:aws:s3:::${BucketName}"
Code language: YAML (yaml)

This is what the aforementioned bucket allows this function to be invokes.

SQS

Resources:
  Queue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub "${Prefix}-Queue"
      ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
      VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)

Queue to which event notifications are sent.

For basic information on SQS, please refer to the following pages.

Check the permissions when the notification destination is SQS.
Messaging S3 event notifications to an SQS queue means that the S3 bucket sends messages to the queue.
This is set by the access policy.

Resources:
  QueuePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties: 
      PolicyDocument:
        Statement: 
          - Action: 
              - sqs:SendMessage
            Condition:
              ArnLike:
                aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}"
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
            Effect: Allow
            Resource: !GetAtt Queue.Arn
            Principal:  
              Service: 
                - s3.amazonaws.com 
      Queues: 
        - !Ref Queue
Code language: YAML (yaml)

The contents of the aforementioned bucket permit messages to be sent to this queue.

This time, we will set up a Lambda function to be automatically triggered when a message is stored in the SQS queue.

Resources:
  EventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Properties: 
      BatchSize: !Ref BatchSize
      Enabled: true
      EventSourceArn: !Ref QueueArn
      FunctionName: !Ref Function2
Code language: YAML (yaml)

For more information, please see the following page

SNS

Resources:
  Topic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub "${Prefix}-Topic"
Code language: YAML (yaml)

The topic of the event notification destination

For basic information about SNS, please refer to the following pages.

No special settings are made.

Check the permissions when the notification destination is SNS.
Messaging S3 event notifications to SNS topics means that the S3 bucket sends messages to the topic.
This is set by the access policy.

Resources:
  TopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties: 
      PolicyDocument:
        Statement: 
          - Action: 
              - sns:Publish
            Condition:
              ArnLike:
                aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}"
              StringEquals:
                aws:SourceAccount: !Ref AWS::AccountId
            Effect: Allow
            Resource: !Ref Topic
            Principal:  
              Service: 
                - s3.amazonaws.com 
      Topics: 
        - !Ref Topic
Code language: YAML (yaml)

Topic subscriptions specify Lambda functions.

Resources:
  TopicSubscription:
    Type: AWS::SNS::Subscription
    Properties: 
      Endpoint: !GetAtt Function3.Arn
      Protocol: lambda
      TopicArn: !Ref SNSTopicArn
Code language: YAML (yaml)

For details, please refer to the following page.

EventBridge

Resources:
  EventsRule:
    Type: AWS::Events::Rule
    Properties: 
      EventBusName: !Ref EventBusName
      EventPattern:
        source:
          - aws.s3
      Name: !Sub "${Prefix}-EventsRule"
      State: ENABLED
      Targets: 
        - Arn: !Ref Function4Arn
          Id: !Ref Function4
Code language: YAML (yaml)

Create an EventBridge rule.

For basic information on EventBridge rules, please see the following pages.

This time, we have a rule where the value of source is “aws.s3” and send events matching this to the target Lambda function.

When a Lambda function is specified as the target, the behavior is that EventBridge invokes the function.
Therefore, it is necessary to authorize EventBridge to invoke the function.

Resources:
  EventsRulePermission: 
    Type: AWS::Lambda::Permission
    Properties: 
      FunctionName: !Ref Function4
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt EventsRule.Arn
Code language: YAML (yaml)

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create a CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • S3 bucket: fa-113
  • SQS Queue: fa-113-Queue
  • SNS Topic: fa-113-Topic
  • EventBridge rule: fa-113-EventsRule
  • Lambda Function 1: fa-113-Function1

Check each resource from the AWS Management Console.

Check the S3 bucket.

Detail of S3 1.

You can see that four notification destinations have been activated.

Check SQS.

Detail of SQS 1.

The SQS queue has been successfully created.
We can also see that function 2 is specified as the Lambda trigger.
This means that when an event notification is sent from the S3 bucket to this SQS queue, function 2 is automatically executed.

Check the SNS.

Detail of SNS 1.

The SNS topic has been successfully created.
You can see that Lambda function 3 is specified as a subscriber to the topic.
This means that when an event notification is sent from the S3 bucket to this SNS topic, function 3 is automatically executed.

Check EventBridge.

Detail of EventBridge 1.
Detail of EventBridge 2.

The EventBridge rule has been successfully created.
You can see that Lambda function 4 is specified as the target of the rule.
This means that when an event matching the rule is sent from the S3 bucket, function 4 is automatically executed.

operation check

Now that we are ready, we will test S3 event notifications.

Event notification to Lambda function

Notify the Lambda function of the event.
This time, the condition for notifying the Lambda function is “s3:ObjectCreated:*”, so place the object.

Detail of S3 2.

Function 1 should be automatically executed when the object is placed.

Check the execution log of function 1.

Detail of Lambda 1.

Indeed, function 1 is executed.
Looking at the eventName value, we can see “ObjectCreated:Put”.
In this way, a Lambda function can be automatically executed upon receiving an S3 event notification.

Event notification to SQS

Notify the SQS queue of the event.
This time, the condition for notification to the queue is “s3:ObjectTagging:*”, so set a tag on the object.

Detail of S3 3.

A tag with the key “foo” and the value “bar” is set.

A message should have been sent to the SQS queue in response.
And since we have set a Lambda trigger on this queue, Lambda function 2 should be executed.

Check the execution log of function 2.

Detail of Lambda 2.

Indeed, function 2 is executed.
Looking at the eventName value, we see “ObjectTagging:Put” and looking at the Event value, we see “s3:TestEvent”.
In this way, we can receive S3 event notifications and automatically notify messages to the SQS queue.

The official AWS explanation of “s3:TestEvent” is as follows

When the notification is first enabled, an s3:TestEvent occurs.

Using Amazon SQS, Amazon SNS, and Lambda

Event notification to SNS

Notify the SNS topic of the event.
This time, the condition for notifying the topic is “s3:ObjectRemoved:*”, so the object is deleted.

Detail of S3 4.

A Object has been deleted.

A message should have been sent to the SNS topic in response.
And since Lambda3 is set as the subscription for this topic, Lambda function 3 should be executed.

Check the execution log for function 3.

Detail of Lambda 3.

Indeed, function 3 is executed.
Looking at the value of eventName, we can see “ObjectRemoved:Delete.
In this way, you can receive S3 event notifications and automatically notify messages to SNS topics.

Event notification to EventBridge

Event notification to EventBridge.
This time, we have enabled notification to EventBridge, so we should be notified each time an operation has been performed so far (adding an object, setting a tag, deleting an object).
And since we have set Lambda function 4 as the target of the EventBridge rule, Lambda function 4 should be executed three times.

Detail of Lambda 4.
Detail of Lambda 5.

Indeed, function 4 is executed.
Looking at the value of detail-type, we can see “Object Created”, “Object Tags Added”, and “Object Deleted”.
In this way, we can receive S3 event notifications and automatically notify messages to EventBridge.

Summary

Four destinations for S3 event notifications have been identified.

タイトルとURLをコピーしました