Configuration to obtain VPC Flow Logs
Check the VPC Flow Logs.
VPC Flow Logs is a service that allows you to obtain traffic information within a VPC.
VPC Flow Logs is a feature that enables you to capture information about the IP traffic going to and from network interfaces in your VPC. Flow log data can be published to Amazon CloudWatch Logs or Amazon S3.
VPC Flow Logs
As mentioned above, you can choose to output the VPC Flow Logs to CloudWatch Logs or S3, but this time we will output the same content to both.
Environment
Prepare two VPC Flow Logs. Configure each one to retrieve traffic information within the above subnets, and specify an S3 bucket and a CloudWatch Logs log group as its output destination.
To verify the VPC Flow Logs, ping one instance to the other and capture the traffic at the subnet level.
CloudFormation template files
We will build the above configuration using CloudFormation.
Place the CloudFormation template at the following URL.
https://github.com/awstut-an-r/awstut-fa/tree/main/014
Template files points
We will cover the key points of each template file to configure this environment.
Configuring VPC Flow Logs – S3 Bucket
Define the VPC Flow Logs related resources in fa-014-flowlog.yaml.
First, we will check how to save the VPC Flow Logs in the S3 bucket.
Resources:
FlowLogBucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
BucketName: !Ref Prefix
FlowLogToS3:
Type: AWS::EC2::FlowLog
DependsOn:
- FlowLogBucket
Properties:
LogDestination: !GetAtt FlowLogBucket.Arn
LogDestinationType: s3
ResourceId: !Ref PrivateSubnet
ResourceType: Subnet
TrafficType: ALL
Code language: YAML (yaml)No special configuration is required for creating an S3 bucket. This time, we will only configure the bucket name and ACL.
Next, let’s check the VPC Flow Logs.
Specify the destination of the log in the LogDestinationType and LogDestination properties. The VPC Flow Logs here is saved in the S3 bucket mentioned above, so specify “s3” in the former and the ARN of the bucket in the latter.
Specify the ResourceType and ResourceId properties, and the target to acquire the VPC Flow Logs.
The former specifies the type of target to acquire the VPC Flow Logs.
The type of resource for which to create the flow log. For example, if you specified a VPC ID for the ResourceId property, specify VPC for this property.
Allowed values: NetworkInterface | Subnet | VPC
AWS::EC2::FlowLog
In this case, we will retrieve the logs of the subnet where the instance is installed, so specify “Subnet”.
The latter specifies the resource ID of the target to be retrieved.
The ID of the subnet, network interface, or VPC for which you want to create a flow log.
AWS::EC2::FlowLog
In this case, we will get the log of the subnet where the instance is installed, so use the built-in function Fn::Ref to specify the ID of the subnet.
TrafficType property specifies the type of traffic to be acquired.
The type of traffic to log. You can log traffic that the resource accepts or rejects, or all traffic.
Allowed values: ACCEPT | ALL | REJECT
AWS::EC2::FlowLog
This time, since we are going to verify the VPC Flow Logs, we will acquire all logs and specify “ALL”.
Configuring VPC Flow Logs – CloudWatch Logs
Then we will check how to save the VPC Flow Logs in CloudWatch Logs.
Resources:
FlowLogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "${Prefix}-FlowLogGroup"
FlowLogToCloudWatchLogs:
Type: AWS::EC2::FlowLog
DependsOn:
- FlowLogGroup
Properties:
DeliverLogsPermissionArn: !GetAtt DeliverLogRole.Arn
LogDestinationType: cloud-watch-logs
LogGroupName: !Sub "${Prefix}-FlowLogGroup"
ResourceId: !Ref PrivateSubnet
ResourceType: Subnet
TrafficType: ALL
DeliverLogRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- vpc-flow-logs.amazonaws.com
Policies:
- PolicyName: DeliverToCloudWatchLogPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- logs:DescribeLogGroups
- logs:DescribeLogStreams
Resource: "*"
Code language: YAML (yaml)No special settings are required to create a log group for CloudWatch Logs. This time, we will only set the name of the log group.
Specify the destination of the log in the LogDestinationType and LogGroupName properties. The VPC Flow Logs here will be saved in CloudWatch Logs as described above, so specify “cloud-watch-logs” in the former and the name of the log group in the latter.
In the DeliverLogsPermissionArn property, specify the IAM role that contains the permissions required to deliver VPC Flow Logs to the log group.
This is the permission required to deliver VPC Flow Logs. I created the IAM role by referring to IAM roles for publishing flow logs to CloudWatch Logs.
Architecting
We will use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and checkresources in stacks
Create a CloudFormation stack.
For more information on how to create stacks and check each stack, please refer to the following page.
After checking the resources for each stack, the information for the main resource created this time is as follows
- ID of Instance1: i-095751cafaf667e27
- ID of Instance2: i-0a0f8ccf1ddd2eead
- IP address of Instance1: 10.0.1.22
- IP address of Instance2: 10.0.1.25
- ID of the ENI for Instance1: eni-01d89470fafe7a86b
- Instance2’s ENI ID: eni-0e61c483ca317dd54
- S3 bucket name: fa-014
- CloudWatch Logs log group name: fa-014-FlowLogGroup
- ID of subnet: subnet-0d3d1285e6b822ae1
Check the creation status of the VPC flow log in the AWS Management Console.
You can see that there are indeed two VPC flow logs set up on the subnet.
Prep – Accessing instance via SSM Session Manager
Now that we are ready, let’s access the instances.
Use the SSM Session Manager to access Instance1.
$ aws ssm start-session \
--target i-095751cafaf667e27
Starting session with SessionId: root-0ab5acb07686fac90
sh-4.2$
Code language: Bash (bash)I was able to access it successfully.
For more information about SSM Session Manager, please refer to the following
Next, ping Instance2 (10.0.1.25).
sh-4.2$ ping 10.0.1.25
PING 10.0.1.25 (10.0.1.25) 56(84) bytes of data.
64 bytes from 10.0.1.25: icmp_seq=1 ttl=255 time=0.124 ms
64 bytes from 10.0.1.25: icmp_seq=2 ttl=255 time=0.157 ms
64 bytes from 10.0.1.25: icmp_seq=3 ttl=255 time=0.134 ms
64 bytes from 10.0.1.25: icmp_seq=4 ttl=255 time=0.160 ms
...
Code language: Bash (bash)Instance2 has responded. We are now ready to go.
Verification 1 – Check VPC Flow Logs in S3 bucket
Check the VPC Flow Logs that was delivered to the S3 bucket.
In this case, the log file has been placed in the S3 bucket, as shown in the following image.
As you can see, VPC Flow Logs files are stored in gzip compressed format.
Download one and check the contents.
Here is an example
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
...
2 [account-id] eni-0e61c483ca317dd54 10.0.1.22 10.0.1.25 0 0 1 57 4788 1641597879 1641597903 ACCEPT OK
2 [account-id] eni-0e61c483ca317dd54 10.0.1.25 10.0.1.22 0 0 1 57 4788 1641597879 1641597903 ACCEPT OK
...Code language: CSS (css)You can read more about how to view the VPC Flow Logs in Flow log records, but there is a record of two instances with the values of “srcaddr” and “dstaddr”, and the “action” is “accept”. You can see that the ping interactions of both instances are successfully recorded in the VPC Flow Logs delivered to the S3 bucket.
Verification 2 – Check VPC Flow Logs in CloudWatch Logs
Next, we will check the VPC Flow Logs delivered to CloudWatch Logs.
First, we will check the status of the log group.
You can see that a stream has been created for each ENI in the log group.
Next, let’s check the contents of the stream for the ENI of Instance1.
The view is the same here: you can see that the VPC Flow Logs delivered to CloudWatch Logs is recorded successfully.
Summary
We have confirmed how to get the VPC flow log.
We confirmed that the VPC flow log can be delivered to S3 bucket or CloudWatch Logs, and both can retrieve the same content.