AWS_EN

Deliver VPC Flow Logs to S3/CloudWatch Logs

スポンサーリンク
Deliver VPC Flow Logs to S3 or CloudWatch Logs with CloudFormation AWS_EN
スポンサーリンク
スポンサーリンク

Configuration to obtain VPC Flow Logs

Check the VPC Flow Logs.

VPC Flow Logs is a service that allows you to obtain traffic information within a VPC.

VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to Amazon CloudWatch Logs or Amazon S3.

VPC Flow Logs

As mentioned above, you can choose to output the VPC Flow Logs to CloudWatch Logs or S3, but this time we will output the same content to both.

Environment

Diagram of delivering VPC Flow Logs to S3 or CloudWatch Logs.

Prepare two VPC Flow Logs. Configure each one to retrieve traffic information within the above subnets, and specify an S3 bucket and a CloudWatch Logs log group as its output destination.

To verify the VPC Flow Logs, ping one instance to the other and capture the traffic at the subnet level.

CloudFormation template files

We will build the above configuration using CloudFormation.

Place the CloudFormation template at the following URL.

awstut-fa/014 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Template files points

We will cover the key points of each template file to configure this environment.

Configuring VPC Flow Logs – S3 Bucket

Define the VPC Flow Logs related resources in fa-014-flowlog.yaml.

First, we will check how to save the VPC Flow Logs in the S3 bucket.

Resources: FlowLogBucket: Type: AWS::S3::Bucket Properties: AccessControl: Private BucketName: !Ref Prefix FlowLogToS3: Type: AWS::EC2::FlowLog DependsOn: - FlowLogBucket Properties: LogDestination: !GetAtt FlowLogBucket.Arn LogDestinationType: s3 ResourceId: !Ref PrivateSubnet ResourceType: Subnet TrafficType: ALL
Code language: YAML (yaml)

No special configuration is required for creating an S3 bucket. This time, we will only configure the bucket name and ACL.

Next, let’s check the VPC Flow Logs.

Specify the destination of the log in the LogDestinationType and LogDestination properties. The VPC Flow Logs here is saved in the S3 bucket mentioned above, so specify “s3” in the former and the ARN of the bucket in the latter.

Specify the ResourceType and ResourceId properties, and the target to acquire the VPC Flow Logs.

The former specifies the type of target to acquire the VPC Flow Logs.

The type of resource for which to create the flow log. For example, if you specified a VPC ID for the ResourceId property, specify VPC for this property.

Allowed values: NetworkInterface | Subnet | VPC

AWS::EC2::FlowLog

In this case, we will retrieve the logs of the subnet where the instance is installed, so specify “Subnet”.

The latter specifies the resource ID of the target to be retrieved.

The ID of the subnet, network interface, or VPC for which you want to create a flow log.

AWS::EC2::FlowLog

In this case, we will get the log of the subnet where the instance is installed, so use the built-in function Fn::Ref to specify the ID of the subnet.

TrafficType property specifies the type of traffic to be acquired.

The type of traffic to log. You can log traffic that the resource accepts or rejects, or all traffic.

Allowed values: ACCEPT | ALL | REJECT

AWS::EC2::FlowLog

This time, since we are going to verify the VPC Flow Logs, we will acquire all logs and specify “ALL”.

Configuring VPC Flow Logs – CloudWatch Logs

Then we will check how to save the VPC Flow Logs in CloudWatch Logs.

Resources: FlowLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: !Sub "${Prefix}-FlowLogGroup" FlowLogToCloudWatchLogs: Type: AWS::EC2::FlowLog DependsOn: - FlowLogGroup Properties: DeliverLogsPermissionArn: !GetAtt DeliverLogRole.Arn LogDestinationType: cloud-watch-logs LogGroupName: !Sub "${Prefix}-FlowLogGroup" ResourceId: !Ref PrivateSubnet ResourceType: Subnet TrafficType: ALL DeliverLogRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - vpc-flow-logs.amazonaws.com Policies: - PolicyName: DeliverToCloudWatchLogPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents - logs:DescribeLogGroups - logs:DescribeLogStreams Resource: "*"
Code language: YAML (yaml)

No special settings are required to create a log group for CloudWatch Logs. This time, we will only set the name of the log group.

Specify the destination of the log in the LogDestinationType and LogGroupName properties. The VPC Flow Logs here will be saved in CloudWatch Logs as described above, so specify “cloud-watch-logs” in the former and the name of the log group in the latter.

In the DeliverLogsPermissionArn property, specify the IAM role that contains the permissions required to deliver VPC Flow Logs to the log group.

This is the permission required to deliver VPC Flow Logs. I created the IAM role by referring to IAM roles for publishing flow logs to CloudWatch Logs.

Architecting

We will use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and checkresources in stacks

Create a CloudFormation stack.

For more information on how to create stacks and check each stack, please refer to the following page.

After checking the resources for each stack, the information for the main resource created this time is as follows

  • ID of Instance1: i-095751cafaf667e27
  • ID of Instance2: i-0a0f8ccf1ddd2eead
  • IP address of Instance1: 10.0.1.22
  • IP address of Instance2: 10.0.1.25
  • ID of the ENI for Instance1: eni-01d89470fafe7a86b
  • Instance2’s ENI ID: eni-0e61c483ca317dd54
  • S3 bucket name: fa-014
  • CloudWatch Logs log group name: fa-014-FlowLogGroup
  • ID of subnet: subnet-0d3d1285e6b822ae1

Check the creation status of the VPC flow log in the AWS Management Console.

VPC flow log is configured on the subnet.

You can see that there are indeed two VPC flow logs set up on the subnet.

Prep – Accessing instance via SSM Session Manager

Now that we are ready, let’s access the instances.

Use the SSM Session Manager to access Instance1.

$ aws ssm start-session \ --target i-095751cafaf667e27 Starting session with SessionId: root-0ab5acb07686fac90 sh-4.2$
Code language: Bash (bash)

I was able to access it successfully.

For more information about SSM Session Manager, please refer to the following

Next, ping Instance2 (10.0.1.25).

sh-4.2$ ping 10.0.1.25 PING 10.0.1.25 (10.0.1.25) 56(84) bytes of data. 64 bytes from 10.0.1.25: icmp_seq=1 ttl=255 time=0.124 ms 64 bytes from 10.0.1.25: icmp_seq=2 ttl=255 time=0.157 ms 64 bytes from 10.0.1.25: icmp_seq=3 ttl=255 time=0.134 ms 64 bytes from 10.0.1.25: icmp_seq=4 ttl=255 time=0.160 ms ...
Code language: Bash (bash)

Instance2 has responded. We are now ready to go.

Verification 1 – Check VPC Flow Logs in S3 bucket

Check the VPC Flow Logs that was delivered to the S3 bucket.

In this case, the log file has been placed in the S3 bucket, as shown in the following image.

VPC FlowLogs delivered to S3 are saved in gzipped format.

As you can see, VPC Flow Logs files are stored in gzip compressed format.

Download one and check the contents.

Here is an example

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status ... 2 [account-id] eni-0e61c483ca317dd54 10.0.1.22 10.0.1.25 0 0 1 57 4788 1641597879 1641597903 ACCEPT OK 2 [account-id] eni-0e61c483ca317dd54 10.0.1.25 10.0.1.22 0 0 1 57 4788 1641597879 1641597903 ACCEPT OK ...
Code language: CSS (css)

You can read more about how to view the VPC Flow Logs in Flow log records, but there is a record of two instances with the values of “srcaddr” and “dstaddr”, and the “action” is “accept”. You can see that the ping interactions of both instances are successfully recorded in the VPC Flow Logs delivered to the S3 bucket.

Verification 2 – Check VPC Flow Logs in CloudWatch Logs

Next, we will check the VPC Flow Logs delivered to CloudWatch Logs.

First, we will check the status of the log group.

VPC FlowLogs delivered to CloudWatch will be streamed for each ENI.

You can see that a stream has been created for each ENI in the log group.

Next, let’s check the contents of the stream for the ENI of Instance1.

VPC Flow Logs are also delivered to CloudWatch Logs.

The view is the same here: you can see that the VPC Flow Logs delivered to CloudWatch Logs is recorded successfully.

Summary

We have confirmed how to get the VPC flow log.

We confirmed that the VPC flow log can be delivered to S3 bucket or CloudWatch Logs, and both can retrieve the same content.

タイトルとURLをコピーしました