I recently encountered a challenge on a network that had to do with certain systems on the LAN not being able to connect to the Internet. The weird thing was that some applications on these systems were able to connect to their online servers for updates and the like, but Internet browsing was not working. In this article, I’m going to replicate some part of the problem and we will troubleshoot together.

The network diagram below is a simplified version of the scenario:

We will be creating this network diagram in GNS3, as shown below. You will notice that I have used two separate Ethernet switches in my GNS3 topology; this is just for writing purposes so that it is clearer to you. I could have used one Ethernet switch and configured different VLANs for the ports.

The “To-Internet” cloud is the LAN NIC of my PC. The “Internal-Host” is connected to my virtual machine (VirtualBox) NIC and I have a Windows XP machine running as a virtual machine on that LAN. Between the Internal-Host and the ASA, I’m using the 192.168.56.0/24 subnet. The ASA is connected to my Internet modem on the 192.168.1.0/24 subnet.

As I described above, the Internal-Host cannot open any webpages. Its default gateway is through the ASA and the ASA is connected to the Internet. The relevant configuration on the ASA is shown below:

hostname ASA
!
interface GigabitEthernet0
 nameif outside
 security-level 0
 ip address 192.168.1.10 255.255.255.0
!
interface GigabitEthernet1
 nameif inside
 security-level 100
 ip address 192.168.56.10 255.255.255.0
!
dns domain-lookup outside
dns server-group DefaultDNS
 name-server 8.8.8.8
object network INTERNAL_NETWORK
 subnet 192.168.56.0 255.255.255.0
object-group service ALLOWED_PORTS tcp
 port-object eq www
 port-object eq https
access-list OUTSIDE_IN extended permit udp any any eq domain
access-list INSIDE_OUT extended permit tcp object INTERNAL_NETWORK any object-group ALLOWED_PORTS
!
object network INTERNAL_NETWORK
 nat (inside,outside) dynamic interface
access-group OUTSIDE_IN in interface outside
access-group INSIDE_OUT in interface inside
route outside 0.0.0.0 0.0.0.0 192.168.1.1 1
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
 message-length maximum 512
policy-map global_policy
 class inspection_default
 inspect dns preset_dns_map
 inspect ftp
 inspect h323 h225
 inspect h323 ras
 inspect rsh
 inspect rtsp
 inspect esmtp
 inspect sqlnet
 inspect skinny
 inspect sunrpc
 inspect xdmcp
 inspect sip
 inspect netbios
 inspect tftp
!
service-policy global_policy global

The Internal-Host has an IP address of 192.168.56.12 and its default gateway is the ASA, as shown below. Also, it uses a public DNS server, 8.8.8.8.

To begin your troubleshooting, you have to first verify that the problem exists rather than relying solely on what you were told. To verify the issue, we will open a web browser and attempt to connect to a website.

As you can see, the problem indeed exists. For troubleshooting, ping is your very good friend. We also need to learn to follow the packet: How does the connection flow from the system to the Internet and back? Since we are trying to access the Internet, it means the packet will be sent to the default gateway, so let us see if we have a connection to our default gateway (the ASA).

We can successfully ping our default gateway, meaning we can rule out a cable issue. That’s the only conclusion we can make at this point, because pinging our default gateway does not guarantee that our web connection is also allowed; i.e., ICMP may be allowed but HTTP may be blocked. In short, when you are troubleshooting, draw your conclusions wisely with enough evidence to support them.

The next thing we will do is to make sure our ASA can ping its own default gateway and, if it can, can it go further to access the Internet? From the ASA configuration above, we know that the ASA’s default gateway is 192.168.1.1.

As shown above, we see that not only can the ASA ping its default gateway, it can also ping a public DNS server. With this, we can rule out an issue with our Internet connection. However, if you want to be thorough, you can assume (until proven false) that there may be something wrong with that public DNS server, 8.8.8.8. To test whether this assumption is true or false, we can try to ping a hostname from our ASA. This is permitted by our configuration because we have a name-server configured on the ASA. This is not always the case, depending on your configuration.

So the DNS server is not faulty (You didn’t think so, did you? ;)). (Top 5 DNS issues (and how to resolve them))Before we go back to the Internal-Host, let’s explore one very useful command on the ASA that allows us to simulate traffic flow and see the result. This is the packet-tracer command; we will use it to see if the Internal-Host can connect to an Internet host on HTTP.

The command I entered on the ASA was: packet-tracer input inside tcp 192.168.56.12 1025 8.8.8.8 www. This command tells the ASA to simulate TCP traffic received on its inside interface from host 192.168.56.12 with source port 1025 going to host 8.8.8.8 on destination port www (80). As we can see, the result is allowed, meaning that the ASA is not blocking that connection in any way.

Hint: This command is very useful during troubleshooting exercises on the ASA, so be sure to take some time to read up on it.

Since the ASA is not blocking our web connection, we can now move on in our troubleshooting by going back to the Internal-Host. Let’s try to ping the DNS server from that host.

Hmm, it fails. If you have an ASA in the path of traffic, let it be your Number 1 suspect when ICMP traffic does not go through. Remember that, by default, ICMP is not part of the inspected traffic in the default MPF configuration on the ASA. But that’s not the only issue here. Take a look at the configuration on the ASA again and notice the access-lists configured:

We have two ACLs configured on the ASA: one is applied to the inside interface and the other is applied to the outside interface. As you can see, ICMP is not permitted to flow from the inside interface, i.e., it is not permitted in the INSIDE_OUT access list. For troubleshooting purposes, we can temporarily allow ICMP traffic. I’d apply the following configuration on the ASA:

access-list INSIDE_OUT permit icmp any any
access-list OUTSIDE_IN permit icmp any any

Note: We had to apply it in both directions because of the ICMP return (echo reply) traffic.

So let’s go back to our Internal-Host to try that ping again.

The ping succeeds. At this point, let’s review our findings: The Internet connection is good; the ASA allows web connection from the inside to the outside (this is permitted in the INSIDE_OUT ACL); the Internal-Host can ping both its default gateway and even its DNS server; the Internal-Host cannot open a webpage.

This should give you a hint that there’s something wrong with your name resolution. To test this, we will attempt to open a web page using the IP address of the web server rather than the name. To do this, I know the IP address of the Google server is 41.221.166.106. I know this from the ping I did on my ASA (scroll up). Now, all I need to do is go to the Internal-Host and open a web page to http://41.221.166.106/.

Aha! We have found the problem: DNS. So, if the Internal-Host can ping its DNS server, then the only other thing in the path of the traffic is the ASA (not considering my Internet modem) but the ASA is allowing web connections (as shown from the packet-tracer output); what then is the problem?

Think of the packet flow again: Internal-Host opens a connection to www.google.com; since this hostname is not contained in its host file (local name resolution file), it will attempt to request the IP address of www.google.com from its DNS server. So it sends a DNS request to 8.8.8.8, which has to go through its default gateway, the ASA. If this request is successful, the DNS server on the Internet will reply with the IP address of www.google.com and then the webpage can open. In this path that I have described, there could be only two issues: Either the request is not getting to the DNS server or the DNS server’s reply is not getting back to the Internal-Host. Packet-tracer to the rescue.

I have altered my packet-tracer command a little to simulate DNS traffic instead of HTTP and the verdict is in: Flow is denied by configured rule. Security reasons may require you to configure an ACL for traffic from the inside (trusted) network, but this also means you have to be careful not to deny necessary traffic, in this case, DNS. So, while the ACL allowed HTTP and HTTPS, it does not allow the complementary DNS service. We need to fix this:

access-list INSIDE_OUT permit udp object INTERNAL_NETWORK any eq 53

Now we can try opening our website again.

There you go! The issue has been resolved. As a side note, while troubleshooting, we may have opened up holes in our security rules so you should remember to clean up when done.

Summary

Troubleshooting can be fun; it can also be a pain. Sometimes, the “little” issue cannot justify the hours put in to find it; but, if you knew from the beginning that the issue was little, what’s the need of troubleshooting? J. To make troubleshooting easier, it’s always best to follow a methodology rather than just rattling things here and there. The methodology that I find works well is to follow the traffic or to think like the router/device (I think Scott Morris said this).

I hope you have found this article helpful and I wish you many more troubleshooting hours to build your skill.