Set threshold values for CloudWatch custom metrics and email notification via SNS
The content is related to monitoring and troubleshooting, which is also part of the scope of the AWS DBA.
In the following pages, we have shown how to use Lambda functions and EventBridge to deliver CloudWatch custom metrics on a regular basis.
In this case, we will consider how to set a threshold value for custom metrics and notify by e-mail when the condition is met.
The following is an official AWS reference on how to notify by email.
Amazon CloudWatch uses Amazon SNS to send email. First, create and subscribe to an SNS topic. When you create a CloudWatch alarm, you can add this SNS topic to send an email notification when the alarm changes state.
Setting up Amazon SNS notifications
This page introduces a configuration for email notification when custom metrics meet the conditions according to the above.
Environment
Create a Lambda function.
The function’s action is to push a random number from 0 to 9 as a CloudWatch custom metric.
The EventBridge rule will execute this function periodically.
The execution interval is 1 minute.
Set a CloudWatch alarm to monitor the value of the custom metric.
If this value is 7 or higher twice in a row, an alarm is triggered.
Set SNS topics to be notified of alarms.
Specify an email address as a subscriber to the SNS topic.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-dva/tree/main/05/003
Explanation of key points of template files
This page focuses on using CloudWatch alarms to set threshold values for custom metrics and email notifications via SNS when conditions are met.
For more information on how to use Lambda functions and EventBridge to deliver custom metrics on a regular basis, please see the following pages.
The following is key information for distributing custom metrics.
- Namespace: test
- Metric name: randomnum
- Dimension name: lambda
- Dimension value: dva-05-003
- Value to be delivered: random value between 0 and 9
CloudWatch Alarm
Resources:
Alarm:
Type: AWS::CloudWatch::Alarm
Properties:
ActionsEnabled: true
AlarmActions:
- !Ref SNSTopicArn
AlarmDescription: test alarm
AlarmName: !Ref Prefix
ComparisonOperator: GreaterThanOrEqualToThreshold
Dimensions:
- Name: !Ref CloudWatchMetricDimensionName
Value: !Ref CloudWatchMetricDimensionValue
EvaluationPeriods: 2
InsufficientDataActions:
- !Ref SNSTopicArn
MetricName: !Ref CloudWatchMetricName
Namespace: !Ref CloudWatchMetricNamespace
OKActions:
- !Ref SNSTopicArn
Period: 60
Statistic: Average
Threshold: 7.0
Unit: Count
Code language: YAML (yaml)When there is a change in alarm status, you can receive notifications by specifying SNS topics in the following three properties.
- AlarmActions property: Notification destination when transitioning to ALARM state.
- InsufficientDataActions property: Notification destination when transitioning to INSUFFICIENT_DATA state.
- OKActions property: Notification destination when transitioning to the OK state.
Four properties (ComparisonOperator, EvaluationPeriods, Period, and Threshold) are used to set the conditions for alarming.
In this case, we combine these properties to evaluate the metric value every 60 seconds and shift to the ALARM state when the metric value is 7.0 or higher for two consecutive times.
Three properties (Dimensions, MetricName, Namespace) are used to set the target metrics.
In this case, we will set the same values as when the metrics are delivered by the Lambda function described above.
SNS
Resources:
Topic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Endpoint: !Ref MailAddress
Protocol: email
TopicName: !Ref Prefix
Code language: YAML (yaml)Set up an email address as a subscriber.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create a CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following pages.
https://awstut.com/en/2021/12/11/cloudformations-nested-stack
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- Lambda function: dva-05-003-function
- EventBridge rule: dva-05-003-EventRule
- CloudWatch Alarm: dva-05-003
- SNS Topic: dva-05-003
Email Address Authentication
If you have designated an email address as a subscriber to an SNS topic, you must authenticate that email address.
The following authentication email will be sent to the specified email address.
Press “Confirm subscription” to proceed with the authentication.
The above page will appear, indicating that authentication has been completed.
Resource Acknowledgement
Check each resource from the AWS Management Console.
First, check the SNS topic.
An email address is specified as a subscriber to the SNS topic.
Check the Lambda function.
The function is successfully created.
Check the EventBridge rules.
This one is also created successfully.
The contents of the Lambda function is executed every minute.
Check CloudWatch alarms.
You can see that the alarm conditions are set as specified in the CloudFormation template.
The State shows that it is “Insufficient data”.
The causes of this state are as follows
The INSUFFICIENT_DATA state indicates one of the following
The INSUFFICIENT_DATA state can indicate any of the following:
・An Amazon CloudWatch alarm just started.
Why is my CloudWatch alarm in INSUFFICIENT_DATA state?
・The metric is unavailable.
・The metric parameters, like namespace, metric name, or dimensions, have been misconfigured.
・There’s not enough data for the metric to determine the alarm state.
This time it is because CloudWatch alarms have not been created long enough and the metrics data has not been sufficiently distributed.
The Actions page also confirms that the aforementioned SNS topic is specified as the destination for notifications.
Operation Check
We are ready.
INSUFFICIENT_DATA -> OK
Wait a moment and check the CloudWatch metrics.
Sure enough, custom metrics are being delivered.
It means that the Lambda function is being executed periodically by EventBridge.
CloudWatch alarms also reflect the delivery status of metrics.
It certainly reflects this.
And the status was INSUFFICIENT_DATA earlier, but now it is OK.
This resulted in the following email
It is noted that the status has changed to OK.
In this way, you can receive email notifications via SNS when CloudWatch alarms change.
OK -> ALARM
ALARM条件を満たすために、しばらく待機します。
Two consecutive numbers of 7 or higher were delivered.
This fulfilled the condition to alarm.
Check CloudWatch alarms.
It has indeed changed to ALARM status.
This resulted in the following email
It states that the status has changed to ALARM.
ALARM -> OK
Finally, check the behavior when returning to the OK state.
A value less than 7 was delivered.
CloudWatch alarm status returned to OK.
This resulted in the following email
It is noted that the status has changed to OK.
Summary
We introduced a configuration that uses CloudWatch alarms to send email notifications via SNS when custom metrics meet conditions.