I recently encountered a challenge on a network that had to do with certain systems on the LAN not being able to connect to the Internet. The weird thing was that some applications on these systems were able to connect to their online servers for updates and the like, but Internet browsing was not working. In this article, I’m going to replicate some part of the problem and we will troubleshoot together.
The network diagram below is a simplified version of the scenario:
We will be creating this network diagram in GNS3, as shown below. You will notice that I have used two separate Ethernet switches in my GNS3 topology; this is just for writing purposes so that it is clearer to you. I could have used one Ethernet switch and configured different VLANs for the ports.
The “To-Internet” cloud is the LAN NIC of my PC. The “Internal-Host” is connected to my virtual machine (VirtualBox) NIC and I have a Windows XP machine running as a virtual machine on that LAN. Between the Internal-Host and the ASA, I’m using the 192.168.56.0/24 subnet. The ASA is connected to my Internet modem on the 192.168.1.0/24 subnet.
As I described above, the Internal-Host cannot open any webpages. Its default gateway is through the ASA and the ASA is connected to the Internet. The relevant configuration on the ASA is shown below:
hostname ASA ! interface GigabitEthernet0 nameif outside security-level 0 ip address 192.168.1.10 255.255.255.0 ! interface GigabitEthernet1 nameif inside security-level 100 ip address 192.168.56.10 255.255.255.0 ! dns domain-lookup outside dns server-group DefaultDNS name-server 184.108.40.206 object network INTERNAL_NETWORK subnet 192.168.56.0 255.255.255.0 object-group service ALLOWED_PORTS tcp port-object eq www port-object eq https access-list OUTSIDE_IN extended permit udp any any eq domain access-list INSIDE_OUT extended permit tcp object INTERNAL_NETWORK any object-group ALLOWED_PORTS ! object network INTERNAL_NETWORK nat (inside,outside) dynamic interface access-group OUTSIDE_IN in interface outside access-group INSIDE_OUT in interface inside route outside 0.0.0.0 0.0.0.0 192.168.1.1 1 ! class-map inspection_default match default-inspection-traffic ! ! policy-map type inspect dns preset_dns_map parameters message-length maximum 512 policy-map global_policy class inspection_default inspect dns preset_dns_map inspect ftp inspect h323 h225 inspect h323 ras inspect rsh inspect rtsp inspect esmtp inspect sqlnet inspect skinny inspect sunrpc inspect xdmcp inspect sip inspect netbios inspect tftp ! service-policy global_policy global
The Internal-Host has an IP address of 192.168.56.12 and its default gateway is the ASA, as shown below. Also, it uses a public DNS server, 220.127.116.11.
To begin your troubleshooting, you have to first verify that the problem exists rather than relying solely on what you were told. To verify the issue, we will open a web browser and attempt to connect to a website.
As you can see, the problem indeed exists. For troubleshooting, ping is your very good friend. We also need to learn to follow the packet: How does the connection flow from the system to the Internet and back? Since we are trying to access the Internet, it means the packet will be sent to the default gateway, so let us see if we have a connection to our default gateway (the ASA).
We can successfully ping our default gateway, meaning we can rule out a cable issue. That’s the only conclusion we can make at this point, because pinging our default gateway does not guarantee that our web connection is also allowed; i.e., ICMP may be allowed but HTTP may be blocked. In short, when you are troubleshooting, draw your conclusions wisely with enough evidence to support them.
The next thing we will do is to make sure our ASA can ping its own default gateway and, if it can, can it go further to access the Internet? From the ASA configuration above, we know that the ASA’s default gateway is 192.168.1.1.
As shown above, we see that not only can the ASA ping its default gateway, it can also ping a public DNS server. With this, we can rule out an issue with our Internet connection. However, if you want to be thorough, you can assume (until proven false) that there may be something wrong with that public DNS server, 18.104.22.168. To test whether this assumption is true or false, we can try to ping a hostname from our ASA. This is permitted by our configuration because we have a name-server configured on the ASA. This is not always the case, depending on your configuration.
So the DNS server is not faulty (You didn’t think so, did you? ;)). (Top 5 DNS issues (and how to resolve them))Before we go back to the Internal-Host, let’s explore one very useful command on the ASA that allows us to simulate traffic flow and see the result. This is the packet-tracer command; we will use it to see if the Internal-Host can connect to an Internet host on HTTP.
The command I entered on the ASA was: packet-tracer input inside tcp 192.168.56.12 1025 22.214.171.124 www. This command tells the ASA to simulate TCP traffic received on its inside interface from host 192.168.56.12 with source port 1025 going to host 126.96.36.199 on destination port www (80). As we can see, the result is allowed, meaning that the ASA is not blocking that connection in any way.
Hint: This command is very useful during troubleshooting exercises on the ASA, so be sure to take some time to read up on it.
Since the ASA is not blocking our web connection, we can now move on in our troubleshooting by going back to the Internal-Host. Let’s try to ping the DNS server from that host.
Hmm, it fails. If you have an ASA in the path of traffic, let it be your Number 1 suspect when ICMP traffic does not go through. Remember that, by default, ICMP is not part of the inspected traffic in the default MPF configuration on the ASA. But that’s not the only issue here. Take a look at the configuration on the ASA again and notice the access-lists configured:
We have two ACLs configured on the ASA: one is applied to the inside interface and the other is applied to the outside interface. As you can see, ICMP is not permitted to flow from the inside interface, i.e., it is not permitted in the INSIDE_OUT access list. For troubleshooting purposes, we can temporarily allow ICMP traffic. I’d apply the following configuration on the ASA:
access-list INSIDE_OUT permit icmp any any access-list OUTSIDE_IN permit icmp any any
Note: We had to apply it in both directions because of the ICMP return (echo reply) traffic.
So let’s go back to our Internal-Host to try that ping again.
The ping succeeds. At this point, let’s review our findings: The Internet connection is good; the ASA allows web connection from the inside to the outside (this is permitted in the INSIDE_OUT ACL); the Internal-Host can ping both its default gateway and even its DNS server; the Internal-Host cannot open a webpage.
This should give you a hint that there’s something wrong with your name resolution. To test this, we will attempt to open a web page using the IP address of the web server rather than the name. To do this, I know the IP address of the Google server is 188.8.131.52. I know this from the ping I did on my ASA (scroll up). Now, all I need to do is go to the Internal-Host and open a web page to http://184.108.40.206/.
Aha! We have found the problem: DNS. So, if the Internal-Host can ping its DNS server, then the only other thing in the path of the traffic is the ASA (not considering my Internet modem) but the ASA is allowing web connections (as shown from the packet-tracer output); what then is the problem?
Think of the packet flow again: Internal-Host opens a connection to www.google.com; since this hostname is not contained in its host file (local name resolution file), it will attempt to request the IP address of www.google.com from its DNS server. So it sends a DNS request to 220.127.116.11, which has to go through its default gateway, the ASA. If this request is successful, the DNS server on the Internet will reply with the IP address of www.google.com and then the webpage can open. In this path that I have described, there could be only two issues: Either the request is not getting to the DNS server or the DNS server’s reply is not getting back to the Internal-Host. Packet-tracer to the rescue.
I have altered my packet-tracer command a little to simulate DNS traffic instead of HTTP and the verdict is in: Flow is denied by configured rule. Security reasons may require you to configure an ACL for traffic from the inside (trusted) network, but this also means you have to be careful not to deny necessary traffic, in this case, DNS. So, while the ACL allowed HTTP and HTTPS, it does not allow the complementary DNS service. We need to fix this:
access-list INSIDE_OUT permit udp object INTERNAL_NETWORK any eq 53
Now we can try opening our website again.
There you go! The issue has been resolved. As a side note, while troubleshooting, we may have opened up holes in our security rules so you should remember to clean up when done.
Troubleshooting can be fun; it can also be a pain. Sometimes, the “little” issue cannot justify the hours put in to find it; but, if you knew from the beginning that the issue was little, what’s the need of troubleshooting? J. To make troubleshooting easier, it’s always best to follow a methodology rather than just rattling things here and there. The methodology that I find works well is to follow the traffic or to think like the router/device (I think Scott Morris said this).
I hope you have found this article helpful and I wish you many more troubleshooting hours to build your skill.