In this article we will see how we can use the VPC Flow Logs feature to monitor the traffic from our Virtual Private Cloud.
VPC Flow Logs is a feature that allows the user to capture information about traffic going in and out through the network interfaces. The log data is stored using Amazon CloudWatch logs.
VMware Training – Resources (Intense)
The purpose of this feature is to help the user troubleshoot traffic issues and understand why resources within a VPC are not accessible.
A flow log can be enabled for a VPC, a subnet or a network interface. The monitoring can be done from the least specific to the most specific. If flow log is enabled for VPC, then all the network interfaces are monitored. The monitoring can be enabled for accepted traffic, rejected traffic or both. The flow log is published to the Amazon CloudWatch as a log group and each network interface is represented as a distinct log stream.
It will take few minutes to generate the data after the flow log is created. Also, if a network interface is added after the flow log was enabled, the corresponding stream will be created as soon as traffic is present on the network interface.
Each log stream has flow records that have specific fields which characterize the traffic recorded. Each record captures the network flow for a specific 5-tuple during a capture window. The 5-tuple is a set of five values that specify the source, destination and the protocol for IP flow. The capture window is a time interval during which the data is aggregated before the flow record is published. This is the flow record format:
version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
The fields are self-explanatory and you can get more information from the references section of this article.
There are also a few limitations about VPC Flow Logs that again are described in detail in another resource from the references section.
So let’s see what my VPC looks like to know what we are going to monitor.
This is the VPC with its CIDR:
There are two subnets, one private and one public, and you can see their CIDRs:
Each subnet is associated to a separate route table. This is the route table to which the public subnet is associated. As you can see, the Internet Gateway is configured and attached to the VPC:
And this is the route table to which the private subnet is associated:
I also have three EC2 instances running: two in the private subnet and one in the public subnet. The EC2 instance from the public subnet has an Elastic IP associated:
As expected, you can connect to the EC2 instances from the private subnet only by first jumping on the EC2 instance from the public subnet and then accessing the EC2 instances from the private subnet. We will test reachability by using the ping utility:
lab@UBUNTU:~/AWS$ ssh -i "AMAZON_LINUX.pem" email@example.com The authenticity of host '18.104.22.168 (22.214.171.124)' can't be established. ECDSA key fingerprint is 0d:33:a6:53:06:0c:ab:1d:d1:c2:67:1f:38:39:80:22. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '126.96.36.199' (ECDSA) to the list of known hosts. __| __|_ ) _| ( / Amazon Linux AMI ___|\___|___| https://aws.amazon.com/amazon-linux-ami/2015.03-release-notes/ No packages needed for security; 1 packages available Run "sudo yum update" to apply all updates. [ec2-user@ip-172-16-0-121 ~]$ [ec2-user@ip-172-16-0-121 ~]$ ping -c 1 172.16.1.196 PING 172.16.1.196 (172.16.1.196) 56(84) bytes of data. 64 bytes from 172.16.1.196: icmp_seq=1 ttl=64 time=0.999 ms --- 172.16.1.196 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 1ms rtt min/avg/max/mdev = 0.999/0.999/0.999/0.000 ms [ec2-user@ip-172-16-0-121 ~]$ [ec2-user@ip-172-16-0-121 ~]$ ping -c 1 172.16.1.197 PING 172.16.1.197 (172.16.1.197) 56(84) bytes of data. 64 bytes from 172.16.1.197: icmp_seq=1 ttl=64 time=1.12 ms --- 172.16.1.197 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 1ms rtt min/avg/max/mdev = 1.127/1.127/1.127/0.000 ms [ec2-user@ip-172-16-0-121 ~]$
To create a flow log, you need to select the VPC for which you want to do this and then right click and select “Create Flow Log”:
Then you will be requested to fill in some details in the next form:
As mentioned, you can filter the traffic to be monitored to those accepted, rejected or both. You will also need to provide the ARN of the IAM role that has the permissions to publish the flow log to the CloudWatch log group. Finally, you need to provide the name of the log group in CloudWatch where the flow logs will be published.
If you don’t have the IAM role created, you can simply click on “Set Up Permissions” and you will be redirected to a page where you can allow access. You can expand the policy document section to see how the policy looks like:
This is how my flow log configuration currently looks like:
Once the flow log is created, you can see it in the “Flow Logs” tab of the VPC dashboard:
If CloudWatch console is checked, you will see that the log group was created:
Selecting the log group, you will see different log streams created, one for each network interface from each EC2 instance from the VPC where you enabled Flow Log:
These are the network interfaces from the three EC2 instances running in the VPC. This is taken from the EC2 console:
If you select one of the log streams, you will see details about the traffic that went in and out through the network interface. This is the log stream generated for an EC2 instance from the private subnet. As you can see, the traffic from 172.16.0.121 (the EC2 instance from the public subnet) was accepted (I generated some SSH and ICMP traffic):
I launched another EC2 instance (the public IP address is 188.8.131.52) in another VPC and then tried to access port 22 and port 80 of the EC2 instance from the public subnet. Port 22 is allowed in the security group, but port 80 is not; hence, you will see some accepted traffic and rejected traffic from the newly launched EC2 instance:
You can also apply complex filters. To see the rejected traffic on port 80 (regardless of whether it is a source or destination port):
And this is pretty much everything about the VPC Flow Logs feature. You can now monitor what kind of traffic is hitting your EC2 instance within a VPC.
It would be useful if there are tools that can analyse the data generated and allow a more human readable display of the data, such as through graphs and tables, but for now, there are none so it’s up to the user to build custom tools.