AWS_EN

Subscription filter to extract errors in Fargate container logs and notify by email

スポンサーリンク
Subscription filter to extract errors in Fargate container logs and notify by email. AWS_EN
スポンサーリンク
スポンサーリンク

Use CloudWatch Logs subscription filter to extract errors from Fargate container logs and notify by email

One of the features of CloudWatch Logs is the subscription filter.
By using the subscription filter, you can extract logs delivered to CloudWatch Logs that contain a specific string and perform an action on them.

In this case, we will deliver logs of Fargate type ECS containers to CloudWatch Logs, extract error logs from them, and aim to notify the contents via email.

Environment

Diagram of Subscription filter to extract errors in Fargate container logs and notify by email.

Create a Fargate type ECS on a private subnet.

Create a VPC endpoint for Logs on the container subnet to deliver logs to CloudWatch Logs.

Enable subscription filtering in CloudWatch Logs.
Create a filter to detect the string “error”.
Create a Lambda function as a resource to associate with the subscription filter.

The function of the Lambda function is to publish a message to the SNS topic.
The function publishes the contents of the logs extracted by the subscription filter.

Place a NAT gateway on a public subnet to retrieve an official Nginx image from DockerHub.

Create an EC2 instance.
Use it as a client to access the container.

CloudFormation template files

Build the above configuration with CloudFormation.
The CloudFormation templates are located at the following URL

awstut-fa/066 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of template files

For basic information on ECS (Fargate), please refer to the following page

For information on how to deploy Fargate on a private subnet, please refer to the following page

SNS Topic

Resources: Topic: Type: AWS::SNS::Topic Properties: Subscription: - Endpoint: !Ref MailAddress Protocol: email TopicName: !Ref Prefix
Code language: YAML (yaml)

The Subscription property is the key.
To specify an email address as a subscriber, specify “email” in the Protocol property and the email address in the Endpoint property.

For details on how to specify an email address as an SNS subscriber, please refer to the following page.

CloudWatch Logs Subscription Filter

In order to send the content detected by the subscription filter to SNS, four types of resources must be created

  • Log Groups
  • Subscription filter
  • Permissions
  • Lambda functions

Log Group

Resources: LogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${Prefix}-LogGroup"
Code language: YAML (yaml)

No special configuration is required.
Simply create a log group.

Subscription Filter

Resources: SubscriptionFilter: Type: AWS::Logs::SubscriptionFilter Properties: DestinationArn: !GetAtt Function.Arn FilterPattern: error LogGroupName: !Ref LogGroup
Code language: YAML (yaml)

The DestinationArn property specifies the AWS resource to be linked when the log is detected.
In this case, specify the Lambda function described below.

Specify the log to be extracted with the FIlterPattern property.
In this case, set “error” to be detected.

Specify the log group for which the subscription filter is enabled in the LogGroupName property.
Specify the aforementioned log group.

Permission

Resources: SubscriptionFilterPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !Ref Function Principal: !Sub "logs.${AWS::Region}.amazonaws.com" SourceArn: !GetAtt LogGroup.Arn
Code language: YAML (yaml)

When the subscription filter finds a log that meets the criteria, the log group is granted permission to invoke the Lambda function described below.

Lambda Function

Resources: Function: Type: AWS::Lambda::Function Properties: Architectures: - !Ref Architecture Environment: Variables: REGION: !Ref AWS::Region TOPIC: !Ref TopicArn Code: ZipFile: | import base64 import boto3 import gzip import json import os topic = os.environ['TOPIC'] region = os.environ['REGION'] client = boto3.client('sns', region_name=region) subject = 'Error Detection.' def lambda_handler(event, context): subscription_data = event['awslogs']['data'] subscription_data_decoded = base64.b64decode(subscription_data) subscription_data_decompressed = gzip.decompress(subscription_data_decoded) subscription_data_loaded = json.loads(subscription_data_decompressed) response = client.publish( TopicArn=topic, Subject=subject, Message=subscription_data_loaded['logEvents'][0]['message'] ) FunctionName: !Sub "${Prefix}-function" Handler: !Ref Handler Runtime: !Ref Runtime Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)

The Environment property allows you to define environment variables that can be passed to the function.
The ARN of the SNS topic mentioned above and the region where the topic was created can be passed.

Define the code to be executed by the Lambda function in inline notation.
For more information, please refer to the following page

The code is based on the following page

CloudWatch Logsの特定文字を検知してログ内容を通知するLambda Function | DevelopersIO
CloudWatch Logsにサブスクリプションフィルター設定して、特定文字を含むログデータをLambda FunctionつかってChatworkに通知しました。

Incidentally, the IAM role for the function is as follows

Resources: FunctionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - lambda.amazonaws.com ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: SNSPublishPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - sns:Publish Resource: - !Ref TopicArn
Code language: YAML (yaml)

First, specify the AWS administrative policy AWSLambdaBasicExecutionRole and grant the necessary permissions to invoke the function.
In addition, grant the permission to publish messages to the SNS topic.

(Reference) Task definition

Resources: TaskDefinition: Type: AWS::ECS::TaskDefinition Properties: ContainerDefinitions: - Name: !Sub "${Prefix}-container" Image: nginx:latest LogConfiguration: LogDriver: awslogs Options: awslogs-group: !Ref LogGroup awslogs-region: !Ref AWS::Region awslogs-stream-prefix: !Sub "${Prefix}-container" Cpu: !Ref TaskCpu ExecutionRoleArn: !Ref FargateTaskExecutionRole Memory: !Ref TaskMemory NetworkMode: awsvpc RequiresCompatibilities: - FARGATE TaskRoleArn: !Ref TaskRole
Code language: YAML (yaml)

The LogConfiguration property of the task definition is the key.
Set the log groups, etc. to be distributed.

For more information on how to distribute ECS logs to CloudWatch Logs, please also check the following page.

Architecting

Use CloudFormation to build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

  • SNS topic: fa-066
  • ECS cluster: fa-066-cluster
  • ECS service: fa-066-service
  • CloudWatch Logs log group: fa-066-LogGroup
  • EC2 instance: i-0bac8729dfcf0dc9f

Authentication of email address

If you have specified an email address as a subscriber to an SNS topic, you must authenticate that email address.

For details, please refer to the following page.

Resource Confirmation

Check each resource from the AWS Management Console.

Check the SNS topic.

Detail of SNS Topic.

You can see that the SNS topic has been successfully created.

In addition, you can see that the email address registered as a subscriber is registered.
The Status value of the email address is “Confirmed,” indicating that the authentication has been completed.

Next, check ECS (Fargate).

Detail of ECS(Fargate) 1.
Detail of ECS(Fargate) 2.

The ECS Cluster Service task has been successfully created.
We can also see that the private address assigned to the task is “10.0.3.61”.

Check the Lambda function.

Detail of Lambda Function.

It is successfully created.
When the subscription filter detects “error”, this function is executed and the log contents are linked to the SNS topic.

Check the CloudWatch Logs log group.

Detail of CloudWatch Logs Log Group 1.
Detail of CloudWatch Logs Log Group 2.

You can see that a stream and a subscription filter have been created in the log group.

Checking Action

Now that everything is ready, access the EC2 instance.

% aws ssm start-session --target i-0bac8729dfcf0dc9f Starting session with SessionId: root-079cb8e3e83a3c9e9 sh-4.2$
Code language: JavaScript (javascript)

For more information on SSM Session Manager, please refer to the following page

Access the container in the task using the curl command.

sh-4.2$ curl http://10.0.3.61 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
Code language: Bash (bash)

The page was successfully accessed.

Next, access a page that does not exist and intentionally generate an error.

sh-4.2$ curl http://10.0.3.61/hogehoge.html <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.23.1</center> </body> </html>
Code language: Bash (bash)

An error occurred.

An email was immediately sent to the registered address.

Authentication of email addresses as notification recipients for SNS.

The body of the email is the content of the error that just occurred.
By using the subscription filter in this way, specific character strings can be detected and email notifications can be sent in conjunction with SNS.

Incidentally, the following is the stream content of the CloudWatch Logs log group.

Error log in CloudWatch Logs.

You can indeed see the original log of the body of the email.

Summary

We have confirmed how to distribute ECS container logs to CloudWatch Logs, extract error logs from them, and notify the contents via email.

タイトルとURLをコピーしました