Welcome to the third article in this troubleshooting series, where we are presented with different troubleshooting scenarios and asked to resolve the issues. The scenario in this lab will focus on sub-sections 7.4 and 7.8 of the CCNA R&S exam topics, which cover IP routing and access lists troubleshooting.
We will be using the following setup for this troubleshooting scenario:
The following access policy has been defined for the network shown above and it has been implemented using ACLs on R3:
Permit SSH connections from networks 192.168.10.0/24 and 192.168.11.0/24 to network 172.16.4.0/24.
Permit ICMP (ping) from networks 192.168.10.0/24 and 192.168.11.0/24 to all the 172.16.X.0/24 networks.
Permit all IP traffic from networks 192.168.12.0/24 and 192.168.13.0/24 to networks 172.16.4.0/24, 172.16.5.0/24 and 172.16.6.0/24.
Permit TCP traffic from networks 192.168.12.0/24 and 192.168.13.0/24 to network 172.16.7.0/24.
Deny all other undefined traffic.
Review the configuration on the devices (restrict your troubleshooting to the routers) and ensure that not only is the policy properly configured but that there is IP connectivity between the networks.
Two files are attached to this article:
trblsht_routing_acl_init.pkt: This Packet Tracer file contains the lab setup with various configuration issues.
trblsht_routing_acl_final.pkt: This Packet Tracer file contains the lab with the issues resolved.
Troubleshooting Scenario Resolution
When ACLs come into the picture, you may need to take another approach to troubleshooting because a traffic flow test case that you want to use may be blocked by the ACL. In this article, we will take the following approach:
Compare the defined access policy to the configured ACLs and make sure the ACLs are properly configured.
Test using various traffic flow cases that should be allowed/denied.
Resolve any issues that arise.
Step 1: Review ACLs
For easy reference, I will paste the ACL configuration on R3 here:
ip access-list extended FROM_R1 permit icmp 192.168.10.0 0.0.1.255 172.16.4.0 0.0.3.255 echo permit tcp 192.168.10.0 0.0.1.255 172.16.4.0 0.0.0.255 eq 22 ip access-list extended FROM_R2 permit ip 192.168.12.0 0.0.1.255 172.16.4.0 0.0.3.255 permit tcp 192.168.7.0 0.0.0.255 172.16.12.0 0.0.1.255 !
Let’s start with the FROM_R1 ACL. The first access policy defined says that SSH should be allowed from 192.168.10.0/24 and 192.168.11.0/24 to 172.16.4.0/24. The second ACE in the FROM_R1 ACL seems to match this policy:
permit tcp 192.168.10.0 0.0.1.255 172.16.4.0 0.0.0.255 eq 22
Since 192.168.10.0/24 and 192.168.11.0/24 can be summarized as 192.168.10.0/23 (192.168.10.0 0.0.1.255 in wildcard mask notation), then we can confirm that this ACE is correct.
The second defined policy says that ping should be allowed from the 192.168.10.0/24 and 192.168.11.0/24 networks to all the 172.16.X.0/24 networks. There are four networks in the 172.16 range, as follows: 172.16.4.0/24, 172.16.5.0/24, 172.16.6.0/24, and 172.16.7.0/24. We know that these networks can all be summarized as 172.16.4.0/22, which in wildcard mask notation is 172.16.4.0 0.0.3.255. Therefore, the other ACE in the FROM_R1 ACL is also correct:
permit icmp 192.168.10.0 0.0.1.255 172.16.4.0 0.0.3.255 echo
It seems the FROM_R1 ACL is fine, so we can now move on to the next ACL: FROM_R2. The third defined access policy says that all IP traffic from 192.168.12.0/24 and 192.168.13.0/24 should be allowed to 172.16.4.0/24, 172.16.5.0/24, and 172.16.6.0/24. The ACE that seems to match this policy is as follows:
permit ip 192.168.12.0 0.0.1.255 172.16.4.0 0.0.3.255
At first glance, this ACE looks correct but, if we look deeper, we notice something is wrong with the destination network portion of the ACE: It matches 172.16.4.0 0.0.3.255, i.e., 172.16.4.0/24 to 172.16.7.0/24. However, the policy does not include 172.16.7.0/24 as part of the networks for which IP traffic should be allowed. Therefore, we need to edit this ACL as follows:
ip access-list extended FROM_R2 no permit ip 192.168.12.0 0.0.1.255 172.16.4.0 0.0.3.255 permit ip 192.168.12.0 0.0.1.255 172.16.4.0 0.0.1.255 permit ip 192.168.12.0 0.0.1.255 172.16.6.0 0.0.0.255
Finally, the last access policy says that TCP should be allowed from 192.168.12.0/24 and 192.168.13.0/24 to 172.16.7.0/24. The last ACE in the FROM_R2 seems to implement this policy:
permit tcp 192.168.7.0 0.0.0.255 172.16.12.0 0.0.1.255
Looking at this ACE, we see a problem – the source and destination networks are in reverse – therefore, we need to fix it as follows:
ip access-list extended FROM_R2 no permit tcp 192.168.7.0 0.0.0.255 172.16.12.0 0.0.1.255 permit tcp 172.16.12.0 0.0.1.255 192.168.7.0 0.0.0.255
To conclude on the ACLs, we need to check how they are applied:
interface GigabitEthernet0/1 ip address 10.1.13.3 255.255.255.0 ip access-group FROM_R1 out ! interface GigabitEthernet0/2 ip address 10.1.23.3 255.255.255.0 ip access-group FROM_R2 in !
As you can see, the FROM_R1 ACL is applied on R3’s Gi0/1 interface in the outbound direction while FROM_R2 is applied on R3’s Gi0/2 interface in the inbound direction. Think about the traffic flow for a minute and the ACEs contained in the ACLs; are the ACLs applied in the right direction? You should conclude that the FROM_R1 ACL is applied in the wrong direction, i.e., it will apply to traffic leaving the Gi0/1 interface instead of traffic coming in through that interface. Therefore, we need to make the following change:
interface GigabitEthernet0/1 no ip access-group FROM_R1 out ip access-group FROM_R1 in
Step 2: Test Traffic Flow
In this step, we will test our configuration to check that everything is working as it should. One thing you should do while creating test cases is to test not only for traffic that should be permitted but also for traffic that should be denied. This way, you stand a higher chance of knowing whether you have mistakenly allowed something to go through.
The first test we will do is if the hosts in the 192.168.10.0/24 and 192.168.11.0/24 networks can open SSH connections to 172.16.4.0/24. The router in that network has an IP address of 172.16.4.100 with a username of “admin” and password of “cisco”:
The second test we will perform is to ping from these same hosts to any device in the 172.16.X.0/24 network. For brevity sake, I will show just two ping sessions—you can confirm the rest:
Now let’s move to the R2 side. We will test that IP traffic (e.g., ICMP ping) can go from 192.168.12.0/24 and 192.168.13.0/24 to the 172.16.X.0/24 (except 172.16.7.0/24):
If you continue troubleshooting, you will discover that this device can ping its default gateway (R2) and so you should move your troubleshooting to that device. If you issue the show ip route command on R2, you will see this:
Strange-looking routing table, right? Well, maybe that’s because IP routing is not enabled on this router:
We need to enable IP routing using the ip routing command. You may also want to enable IP CEF (ip cef) because it was automatically disabled when routing was disabled.
Now if we check the routing table on R2, we see that it has a default route to R3:
However, if we try to ping again from 172.16.12.100 to 172.16.4.100, it will fail. One thing that helps me during troubleshooting is to turn on ICMP debugging (maybe with an ACL) if possible. So we can turn on ICMP debugging on R3_1 (debug ip icmp) and try to ping again:
Note: Be careful of using the debug command in a production network, as you may actually overwhelm the device.
Think for a minute about how the traffic flowed: 172.16.12.100 sends the traffic to its default gateway (R2); R2 checked its routing table for directions on reaching 172.16.4.100 and sends it to R3; R3 has a direct connection to that network, so it forwards it to R3_1. We know R3_1 received the ping packet because it sends an echo reply as shown in the debug output above. However, the next message tells us the problem: R3 (172.16.4.1) sends an ICMP unreachable to R3_1. Therefore let’s check what is happening on R3:
R3 does not have a route to that network. If we check the entire routing table, you will also discover that R3 doesn’t have a route for 192.168.13.0/24. Let’s add those routes:
ip route 192.168.12.0 255.255.254.0 10.1.23.2
With the routing sorted, we should now have a reply to our ping:
You can try other tests to make sure everything is working as it should.
I hope you have found this lab challenging. In this lab, we have looked at IP routing and ACL troubleshooting.
Binary Math – Part I: http://blog.ine.com/2008/09/15/binary-math-part-i/