Subscription filter to extract errors in Fargate container logs and notify by email

TOC

Use CloudWatch Logs subscription filter to extract errors from Fargate container logs and notify by email

One of the features of CloudWatch Logs is the subscription filter.
By using the subscription filter, you can extract logs delivered to CloudWatch Logs that contain a specific string and perform an action on them.

In this case, we will deliver logs of Fargate type ECS containers to CloudWatch Logs, extract error logs from them, and aim to notify the contents via email.

Environment

Diagram of Subscription filter to extract errors in Fargate container logs and notify by email.

Create a Fargate type ECS on a private subnet.

Create a VPC endpoint for Logs on the container subnet to deliver logs to CloudWatch Logs.

Enable subscription filtering in CloudWatch Logs.
Create a filter to detect the string “error”.
Create a Lambda function as a resource to associate with the subscription filter.

The function of the Lambda function is to publish a message to the SNS topic.
The function publishes the contents of the logs extracted by the subscription filter.

Place a NAT gateway on a public subnet to retrieve an official Nginx image from DockerHub.

Create an EC2 instance.
Use it as a client to access the container.

CloudFormation template files

Build the above configuration with CloudFormation.
The CloudFormation templates are located at the following URL

https://github.com/awstut-an-r/awstut-fa/tree/main/066

Explanation of key points of template files

For basic information on ECS (Fargate), please refer to the following page

For information on how to deploy Fargate on a private subnet, please refer to the following page

SNS Topic

Resources:
  Topic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: !Ref MailAddress
          Protocol: email
      TopicName: !Ref Prefix
Code language: YAML (yaml)

The Subscription property is the key.
To specify an email address as a subscriber, specify “email” in the Protocol property and the email address in the Endpoint property.

For details on how to specify an email address as an SNS subscriber, please refer to the following page.

CloudWatch Logs Subscription Filter

In order to send the content detected by the subscription filter to SNS, four types of resources must be created

Log Groups
Subscription filter
Permissions
Lambda functions

Log Group

Resources:
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "${Prefix}-LogGroup"
Code language: YAML (yaml)

No special configuration is required.
Simply create a log group.

Subscription Filter

Resources:
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: !GetAtt Function.Arn
      FilterPattern: error
      LogGroupName: !Ref LogGroup
Code language: YAML (yaml)

The DestinationArn property specifies the AWS resource to be linked when the log is detected.
In this case, specify the Lambda function described below.

Specify the log to be extracted with the FIlterPattern property.
In this case, set “error” to be detected.

Specify the log group for which the subscription filter is enabled in the LogGroupName property.
Specify the aforementioned log group.

Permission

Resources:
  SubscriptionFilterPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref Function
      Principal: !Sub "logs.${AWS::Region}.amazonaws.com"
      SourceArn: !GetAtt LogGroup.Arn
Code language: YAML (yaml)

When the subscription filter finds a log that meets the criteria, the log group is granted permission to invoke the Lambda function described below.

Lambda Function

Resources:
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          REGION: !Ref AWS::Region
          TOPIC: !Ref TopicArn
      Code:
        ZipFile: |
          import base64
          import boto3
          import gzip
          import json
          import os

          topic = os.environ['TOPIC']
          region = os.environ['REGION']

          client = boto3.client('sns', region_name=region)
          subject = 'Error Detection.'

          def lambda_handler(event, context):
            subscription_data = event['awslogs']['data']

            subscription_data_decoded = base64.b64decode(subscription_data)

            subscription_data_decompressed = gzip.decompress(subscription_data_decoded)

            subscription_data_loaded = json.loads(subscription_data_decompressed)

            response = client.publish(
              TopicArn=topic,
              Subject=subject,
              Message=subscription_data_loaded['logEvents'][0]['message']
              )
      FunctionName: !Sub "${Prefix}-function"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)

The Environment property allows you to define environment variables that can be passed to the function.
The ARN of the SNS topic mentioned above and the region where the topic was created can be passed.

Define the code to be executed by the Lambda function in inline notation.
For more information, please refer to the following page

The code is based on the following page

https://dev.classmethod.jp/articles/cwl-lambda-sns-publish/

Incidentally, the IAM role for the function is as follows

Resources:
  FunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: SNSPublishPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sns:Publish
                Resource:
                  - !Ref TopicArn
Code language: YAML (yaml)

First, specify the AWS administrative policy AWSLambdaBasicExecutionRole and grant the necessary permissions to invoke the function.
In addition, grant the permission to publish messages to the SNS topic.

(Reference) Task definition

Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Name: !Sub "${Prefix}-container"
          Image: nginx:latest
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Sub "${Prefix}-container"
      Cpu: !Ref TaskCpu
      ExecutionRoleArn: !Ref FargateTaskExecutionRole
      Memory: !Ref TaskMemory
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      TaskRoleArn: !Ref TaskRole
Code language: YAML (yaml)

The LogConfiguration property of the task definition is the key.
Set the log groups, etc. to be distributed.

For more information on how to distribute ECS logs to CloudWatch Logs, please also check the following page.

Architecting

Use CloudFormation to build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

SNS topic: fa-066
ECS cluster: fa-066-cluster
ECS service: fa-066-service
CloudWatch Logs log group: fa-066-LogGroup
EC2 instance: i-0bac8729dfcf0dc9f

Authentication of email address

If you have specified an email address as a subscriber to an SNS topic, you must authenticate that email address.

For details, please refer to the following page.

Resource Confirmation

Check each resource from the AWS Management Console.

Check the SNS topic.

You can see that the SNS topic has been successfully created.

In addition, you can see that the email address registered as a subscriber is registered.
The Status value of the email address is “Confirmed,” indicating that the authentication has been completed.

Next, check ECS (Fargate).

The ECS Cluster Service task has been successfully created.
We can also see that the private address assigned to the task is “10.0.3.61”.

Check the Lambda function.

It is successfully created.
When the subscription filter detects “error”, this function is executed and the log contents are linked to the SNS topic.

Check the CloudWatch Logs log group.

You can see that a stream and a subscription filter have been created in the log group.

Checking Action

Now that everything is ready, access the EC2 instance.

% aws ssm start-session --target i-0bac8729dfcf0dc9f

Starting session with SessionId: root-079cb8e3e83a3c9e9
sh-4.2$Code language: JavaScript (javascript)

For more information on SSM Session Manager, please refer to the following page

Access the container in the task using the curl command.

sh-4.2$ curl http://10.0.3.61
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Code language: Bash (bash)

The page was successfully accessed.

Next, access a page that does not exist and intentionally generate an error.

sh-4.2$ curl http://10.0.3.61/hogehoge.html
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
Code language: Bash (bash)

An error occurred.

An email was immediately sent to the registered address.

Authentication of email addresses as notification recipients for SNS.

The body of the email is the content of the error that just occurred.
By using the subscription filter in this way, specific character strings can be detected and email notifications can be sent in conjunction with SNS.

Incidentally, the following is the stream content of the CloudWatch Logs log group.