AWS_EN

4 destinations for S3 event notifications – SNS/SQS/Lambda/EventBridge

スポンサーリンク
4 destinations for S3 event notifications - SNS/SQS/Lambda/EventBridge AWS_EN
スポンサーリンク
スポンサーリンク

4 destinations for S3 event notifications – SNS/SQS/Lambda/EventBridge

In the following page, we introduced a configuration that automatically creates thumbnail images from uploaded images using the S3 event notification function.

The above page was configured to trigger a Lambda function with an event notification.

This time we will review the four destinations for event notifications.

Environment

Diagram of 4 destinations for S3 event notifications - SNS/SQS/Lambda/EventBridge

Create S3 buckets.
Enable the event notification features.
Specify the following four resources as destinations for notifications.

  • Lambda Function
  • SQS
  • SNS
  • EventBridge

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL.

awstut-fa/113 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of template files

S3 bucket

Resources: Bucket: Type: AWS::S3::Bucket Properties: AccessControl: Private BucketName: !Ref BucketName NotificationConfiguration: LambdaConfigurations: - Event: "s3:ObjectCreated:*" Function: !Ref Function1Arn QueueConfigurations: - Event: "s3:ObjectTagging:*" Queue: !Ref QueueArn TopicConfigurations: - Event: "s3:ObjectRemoved:*" Topic: !Ref TopicArn EventBridgeConfiguration: EventBridgeEnabled: true
Code language: YAML (yaml)

Specify four notification destinations.

The LambdaConfigurations property is used to notify the Lambda function.
The condition for event notification is set to “s3:ObjectCreated:*” and the function will be triggered when an object is placed in the bucket.

The QueueConfigurations property is used to notify SQS.
As for the conditions for event notification, set “s3:ObjectTagging:*” to send a message to the SQS queue when an object in the bucket is tagged (added, etc.).

Notification to SNS is done using the TopicConfigurations property.
The condition for event notification is “s3:ObjectRemoved:*”, which sends a message to the SNS topic when an object is deleted from the bucket.

Notifications to EventBridge use the EventBridgeConfiguration property.
This property enables notification to EventBridge.

Lambda Functions

Resources: Function1: Type: AWS::Lambda::Function Properties: Architectures: - !Ref Architecture Code: ZipFile: | def lambda_handler(event, context): print(event) FunctionName: !Sub "${Prefix}-Function1" Handler: !Ref Handler Runtime: !Ref Runtime Role: !GetAtt Function1Role.Arn
Code language: YAML (yaml)

Lambda function for event notification destination.

The code to be executed by the Lambda function in inline format.
For more information, please refer to the following page.

The code to be executed is a simple one that outputs the contents of the event object.

Incidentally, SQS, SNS, and EventBridge, described below, also execute functions with similar contents.

Check the permissions when the notification is to a Lambda function.
Triggering a Lambda function in an S3 event notification means that the S3 bucket invokes the Lambda function.
This is set by the Lambda resource-based policy.

Resources: S3Permission: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref Function1 Action: lambda:InvokeFunction Principal: s3.amazonaws.com SourceArn: !Sub "arn:aws:s3:::${BucketName}"
Code language: YAML (yaml)

This is what the aforementioned bucket allows this function to be invokes.

SQS

Resources: Queue: Type: AWS::SQS::Queue Properties: QueueName: !Sub "${Prefix}-Queue" ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)

Queue to which event notifications are sent.

For basic information on SQS, please refer to the following pages.

Check the permissions when the notification destination is SQS.
Messaging S3 event notifications to an SQS queue means that the S3 bucket sends messages to the queue.
This is set by the access policy.

Resources: QueuePolicy: Type: AWS::SQS::QueuePolicy Properties: PolicyDocument: Statement: - Action: - sqs:SendMessage Condition: ArnLike: aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}" StringEquals: aws:SourceAccount: !Ref AWS::AccountId Effect: Allow Resource: !GetAtt Queue.Arn Principal: Service: - s3.amazonaws.com Queues: - !Ref Queue
Code language: YAML (yaml)

The contents of the aforementioned bucket permit messages to be sent to this queue.

This time, we will set up a Lambda function to be automatically triggered when a message is stored in the SQS queue.

Resources: EventSourceMapping: Type: AWS::Lambda::EventSourceMapping Properties: BatchSize: !Ref BatchSize Enabled: true EventSourceArn: !Ref QueueArn FunctionName: !Ref Function2
Code language: YAML (yaml)

For more information, please see the following page

SNS

Resources: Topic: Type: AWS::SNS::Topic Properties: TopicName: !Sub "${Prefix}-Topic"
Code language: YAML (yaml)

The topic of the event notification destination

For basic information about SNS, please refer to the following pages.

No special settings are made.

Check the permissions when the notification destination is SNS.
Messaging S3 event notifications to SNS topics means that the S3 bucket sends messages to the topic.
This is set by the access policy.

Resources: TopicPolicy: Type: AWS::SNS::TopicPolicy Properties: PolicyDocument: Statement: - Action: - sns:Publish Condition: ArnLike: aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}" StringEquals: aws:SourceAccount: !Ref AWS::AccountId Effect: Allow Resource: !Ref Topic Principal: Service: - s3.amazonaws.com Topics: - !Ref Topic
Code language: YAML (yaml)

Topic subscriptions specify Lambda functions.

Resources: TopicSubscription: Type: AWS::SNS::Subscription Properties: Endpoint: !GetAtt Function3.Arn Protocol: lambda TopicArn: !Ref SNSTopicArn
Code language: YAML (yaml)

For details, please refer to the following page.

EventBridge

Resources: EventsRule: Type: AWS::Events::Rule Properties: EventBusName: !Ref EventBusName EventPattern: source: - aws.s3 Name: !Sub "${Prefix}-EventsRule" State: ENABLED Targets: - Arn: !Ref Function4Arn Id: !Ref Function4
Code language: YAML (yaml)

Create an EventBridge rule.

For basic information on EventBridge rules, please see the following pages.

This time, we have a rule where the value of source is “aws.s3” and send events matching this to the target Lambda function.

When a Lambda function is specified as the target, the behavior is that EventBridge invokes the function.
Therefore, it is necessary to authorize EventBridge to invoke the function.

Resources: EventsRulePermission: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref Function4 Action: lambda:InvokeFunction Principal: events.amazonaws.com SourceArn: !GetAtt EventsRule.Arn
Code language: YAML (yaml)

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create a CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • S3 bucket: fa-113
  • SQS Queue: fa-113-Queue
  • SNS Topic: fa-113-Topic
  • EventBridge rule: fa-113-EventsRule
  • Lambda Function 1: fa-113-Function1

Check each resource from the AWS Management Console.

Check the S3 bucket.

Detail of S3 1.

You can see that four notification destinations have been activated.

Check SQS.

Detail of SQS 1.

The SQS queue has been successfully created.
We can also see that function 2 is specified as the Lambda trigger.
This means that when an event notification is sent from the S3 bucket to this SQS queue, function 2 is automatically executed.

Check the SNS.

Detail of SNS 1.

The SNS topic has been successfully created.
You can see that Lambda function 3 is specified as a subscriber to the topic.
This means that when an event notification is sent from the S3 bucket to this SNS topic, function 3 is automatically executed.

Check EventBridge.

Detail of EventBridge 1.
Detail of EventBridge 2.

The EventBridge rule has been successfully created.
You can see that Lambda function 4 is specified as the target of the rule.
This means that when an event matching the rule is sent from the S3 bucket, function 4 is automatically executed.

operation check

Now that we are ready, we will test S3 event notifications.

Event notification to Lambda function

Notify the Lambda function of the event.
This time, the condition for notifying the Lambda function is “s3:ObjectCreated:*”, so place the object.

Detail of S3 2.

Function 1 should be automatically executed when the object is placed.

Check the execution log of function 1.

Detail of Lambda 1.

Indeed, function 1 is executed.
Looking at the eventName value, we can see “ObjectCreated:Put”.
In this way, a Lambda function can be automatically executed upon receiving an S3 event notification.

Event notification to SQS

Notify the SQS queue of the event.
This time, the condition for notification to the queue is “s3:ObjectTagging:*”, so set a tag on the object.

Detail of S3 3.

A tag with the key “foo” and the value “bar” is set.

A message should have been sent to the SQS queue in response.
And since we have set a Lambda trigger on this queue, Lambda function 2 should be executed.

Check the execution log of function 2.

Detail of Lambda 2.

Indeed, function 2 is executed.
Looking at the eventName value, we see “ObjectTagging:Put” and looking at the Event value, we see “s3:TestEvent”.
In this way, we can receive S3 event notifications and automatically notify messages to the SQS queue.

The official AWS explanation of “s3:TestEvent” is as follows

When the notification is first enabled, an s3:TestEvent occurs.

Using Amazon SQS, Amazon SNS, and Lambda

Event notification to SNS

Notify the SNS topic of the event.
This time, the condition for notifying the topic is “s3:ObjectRemoved:*”, so the object is deleted.

Detail of S3 4.

A Object has been deleted.

A message should have been sent to the SNS topic in response.
And since Lambda3 is set as the subscription for this topic, Lambda function 3 should be executed.

Check the execution log for function 3.

Detail of Lambda 3.

Indeed, function 3 is executed.
Looking at the value of eventName, we can see “ObjectRemoved:Delete.
In this way, you can receive S3 event notifications and automatically notify messages to SNS topics.

Event notification to EventBridge

Event notification to EventBridge.
This time, we have enabled notification to EventBridge, so we should be notified each time an operation has been performed so far (adding an object, setting a tag, deleting an object).
And since we have set Lambda function 4 as the target of the EventBridge rule, Lambda function 4 should be executed three times.

Detail of Lambda 4.
Detail of Lambda 5.

Indeed, function 4 is executed.
Looking at the value of detail-type, we can see “Object Created”, “Object Tags Added”, and “Object Deleted”.
In this way, we can receive S3 event notifications and automatically notify messages to EventBridge.

Summary

Four destinations for S3 event notifications have been identified.

タイトルとURLをコピーしました