4 destinations for S3 event notifications – SNS/SQS/Lambda/EventBridge
In the following page, we introduced a configuration that automatically creates thumbnail images from uploaded images using the S3 event notification function.
The above page was configured to trigger a Lambda function with an event notification.
This time we will review the four destinations for event notifications.
Environment
Create S3 buckets.
Enable the event notification features.
Specify the following four resources as destinations for notifications.
- Lambda Function
- SQS
- SNS
- EventBridge
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL.
https://github.com/awstut-an-r/awstut-fa/tree/main/113
Explanation of key points of template files
S3 bucket
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Ref BucketName
NotificationConfiguration:
LambdaConfigurations:
- Event: "s3:ObjectCreated:*"
Function: !Ref Function1Arn
QueueConfigurations:
- Event: "s3:ObjectTagging:*"
Queue: !Ref QueueArn
TopicConfigurations:
- Event: "s3:ObjectRemoved:*"
Topic: !Ref TopicArn
EventBridgeConfiguration:
EventBridgeEnabled: true
Code language: YAML (yaml)Specify four notification destinations.
The LambdaConfigurations property is used to notify the Lambda function.
The condition for event notification is set to “s3:ObjectCreated:*” and the function will be triggered when an object is placed in the bucket.
The QueueConfigurations property is used to notify SQS.
As for the conditions for event notification, set “s3:ObjectTagging:*” to send a message to the SQS queue when an object in the bucket is tagged (added, etc.).
Notification to SNS is done using the TopicConfigurations property.
The condition for event notification is “s3:ObjectRemoved:*”, which sends a message to the SNS topic when an object is deleted from the bucket.
Notifications to EventBridge use the EventBridgeConfiguration property.
This property enables notification to EventBridge.
Lambda Functions
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Code:
ZipFile: |
def lambda_handler(event, context):
print(event)
FunctionName: !Sub "${Prefix}-Function1"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt Function1Role.Arn
Code language: YAML (yaml)Lambda function for event notification destination.
The code to be executed by the Lambda function in inline format.
For more information, please refer to the following page.
The code to be executed is a simple one that outputs the contents of the event object.
Incidentally, SQS, SNS, and EventBridge, described below, also execute functions with similar contents.
Check the permissions when the notification is to a Lambda function.
Triggering a Lambda function in an S3 event notification means that the S3 bucket invokes the Lambda function.
This is set by the Lambda resource-based policy.
Resources:
S3Permission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref Function1
Action: lambda:InvokeFunction
Principal: s3.amazonaws.com
SourceArn: !Sub "arn:aws:s3:::${BucketName}"
Code language: YAML (yaml)This is what the aforementioned bucket allows this function to be invokes.
SQS
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: !Sub "${Prefix}-Queue"
ReceiveMessageWaitTimeSeconds: !Ref ReceiveMessageWaitTimeSeconds
VisibilityTimeout: !Ref VisibilityTimeout
Code language: YAML (yaml)Queue to which event notifications are sent.
For basic information on SQS, please refer to the following pages.
Check the permissions when the notification destination is SQS.
Messaging S3 event notifications to an SQS queue means that the S3 bucket sends messages to the queue.
This is set by the access policy.
Resources:
QueuePolicy:
Type: AWS::SQS::QueuePolicy
Properties:
PolicyDocument:
Statement:
- Action:
- sqs:SendMessage
Condition:
ArnLike:
aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}"
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
Effect: Allow
Resource: !GetAtt Queue.Arn
Principal:
Service:
- s3.amazonaws.com
Queues:
- !Ref Queue
Code language: YAML (yaml)The contents of the aforementioned bucket permit messages to be sent to this queue.
This time, we will set up a Lambda function to be automatically triggered when a message is stored in the SQS queue.
Resources:
EventSourceMapping:
Type: AWS::Lambda::EventSourceMapping
Properties:
BatchSize: !Ref BatchSize
Enabled: true
EventSourceArn: !Ref QueueArn
FunctionName: !Ref Function2
Code language: YAML (yaml)For more information, please see the following page
SNS
Resources:
Topic:
Type: AWS::SNS::Topic
Properties:
TopicName: !Sub "${Prefix}-Topic"
Code language: YAML (yaml)The topic of the event notification destination
For basic information about SNS, please refer to the following pages.
No special settings are made.
Check the permissions when the notification destination is SNS.
Messaging S3 event notifications to SNS topics means that the S3 bucket sends messages to the topic.
This is set by the access policy.
Resources:
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Statement:
- Action:
- sns:Publish
Condition:
ArnLike:
aws:SourceArn: !Sub "arn:aws:s3:::${BucketName}"
StringEquals:
aws:SourceAccount: !Ref AWS::AccountId
Effect: Allow
Resource: !Ref Topic
Principal:
Service:
- s3.amazonaws.com
Topics:
- !Ref Topic
Code language: YAML (yaml)Topic subscriptions specify Lambda functions.
Resources:
TopicSubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: !GetAtt Function3.Arn
Protocol: lambda
TopicArn: !Ref SNSTopicArn
Code language: YAML (yaml)For details, please refer to the following page.
EventBridge
Resources:
EventsRule:
Type: AWS::Events::Rule
Properties:
EventBusName: !Ref EventBusName
EventPattern:
source:
- aws.s3
Name: !Sub "${Prefix}-EventsRule"
State: ENABLED
Targets:
- Arn: !Ref Function4Arn
Id: !Ref Function4
Code language: YAML (yaml)Create an EventBridge rule.
For basic information on EventBridge rules, please see the following pages.
This time, we have a rule where the value of source is “aws.s3” and send events matching this to the target Lambda function.
When a Lambda function is specified as the target, the behavior is that EventBridge invokes the function.
Therefore, it is necessary to authorize EventBridge to invoke the function.
Resources:
EventsRulePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref Function4
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventsRule.Arn
Code language: YAML (yaml)Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create a CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- S3 bucket: fa-113
- SQS Queue: fa-113-Queue
- SNS Topic: fa-113-Topic
- EventBridge rule: fa-113-EventsRule
- Lambda Function 1: fa-113-Function1
Check each resource from the AWS Management Console.
Check the S3 bucket.
You can see that four notification destinations have been activated.
Check SQS.
The SQS queue has been successfully created.
We can also see that function 2 is specified as the Lambda trigger.
This means that when an event notification is sent from the S3 bucket to this SQS queue, function 2 is automatically executed.
Check the SNS.
The SNS topic has been successfully created.
You can see that Lambda function 3 is specified as a subscriber to the topic.
This means that when an event notification is sent from the S3 bucket to this SNS topic, function 3 is automatically executed.
Check EventBridge.
The EventBridge rule has been successfully created.
You can see that Lambda function 4 is specified as the target of the rule.
This means that when an event matching the rule is sent from the S3 bucket, function 4 is automatically executed.
operation check
Now that we are ready, we will test S3 event notifications.
Event notification to Lambda function
Notify the Lambda function of the event.
This time, the condition for notifying the Lambda function is “s3:ObjectCreated:*”, so place the object.
Function 1 should be automatically executed when the object is placed.
Check the execution log of function 1.
Indeed, function 1 is executed.
Looking at the eventName value, we can see “ObjectCreated:Put”.
In this way, a Lambda function can be automatically executed upon receiving an S3 event notification.
Event notification to SQS
Notify the SQS queue of the event.
This time, the condition for notification to the queue is “s3:ObjectTagging:*”, so set a tag on the object.
A tag with the key “foo” and the value “bar” is set.
A message should have been sent to the SQS queue in response.
And since we have set a Lambda trigger on this queue, Lambda function 2 should be executed.
Check the execution log of function 2.
Indeed, function 2 is executed.
Looking at the eventName value, we see “ObjectTagging:Put” and looking at the Event value, we see “s3:TestEvent”.
In this way, we can receive S3 event notifications and automatically notify messages to the SQS queue.
The official AWS explanation of “s3:TestEvent” is as follows
When the notification is first enabled, an s3:TestEvent occurs.
Using Amazon SQS, Amazon SNS, and Lambda
Event notification to SNS
Notify the SNS topic of the event.
This time, the condition for notifying the topic is “s3:ObjectRemoved:*”, so the object is deleted.
A Object has been deleted.
A message should have been sent to the SNS topic in response.
And since Lambda3 is set as the subscription for this topic, Lambda function 3 should be executed.
Check the execution log for function 3.
Indeed, function 3 is executed.
Looking at the value of eventName, we can see “ObjectRemoved:Delete.
In this way, you can receive S3 event notifications and automatically notify messages to SNS topics.
Event notification to EventBridge
Event notification to EventBridge.
This time, we have enabled notification to EventBridge, so we should be notified each time an operation has been performed so far (adding an object, setting a tag, deleting an object).
And since we have set Lambda function 4 as the target of the EventBridge rule, Lambda function 4 should be executed three times.
Indeed, function 4 is executed.
Looking at the value of detail-type, we can see “Object Created”, “Object Tags Added”, and “Object Deleted”.
In this way, we can receive S3 event notifications and automatically notify messages to EventBridge.
Summary
Four destinations for S3 event notifications have been identified.