This article will focus on troubleshooting various LAN problems that you might run into when dealing with Cisco switches.
Before you can start to troubleshoot anything you have to have the theoretical knowledge about that specific technology and know what would be the expected behavior.
CCNA Training – Resources (Intense)
Because this article will show you how you can troubleshoot VLANs, VTP (VLAN trunk protocol), DTP (dynamic trunking protocol), Etherchannel and port security, I advise you to revise the very good information that you can find on these links : All you have to know about VLANs for the Cisco CCNA Exam (Part 1), All you have to know about VLANs for the Cisco CCNA Exam (part 2) and Configuring LAN switches.
I will use only “show” commands to find the problems that keep the hosts from communicating with each other. The configuration will be checked only to confirm of the configuration errors.
The topology is described below:
Host-1 and Host-2 are in VLAN 10 and Host-3 and Host-4 are in VLAN 20. Our goal is to have communication between hosts in the same VLAN. Another constraint is to have communication between the hosts in the same VLAN even if the direct link between the switches where the hosts are connected is unavailable (the link has problems and cannot be used). For instance, if the link between SW-1 and SW-2 is down, Host-1 should reach Host-2 using the path SW-1 – SW-3 – SW-2.
So, let’s start by checking the communication in VLAN 10 between Host-1 and Host-2 by pinging Host-2 from Host-1:
We don’t have reachability; it seems that the ICMP packets are not received by Host-2. Or maybe they are received, but the returning packets are not received by Host-1.
The first step in this scenario will be to make sure that the MAC address of Host-1 is known by SW-1 and SW-2 and the MAC address of Host-2 is known by SW-1 and SW-2.
Let’s check if MAC address of Host-1 is present in the MAC address table of SW-1 and SW-2:
Let’s check the presence of Host-1 MAC address in SW-2:
We don’t have the MAC address. Now we know why there is no communication between the two hosts.
At this moment, one option that we can think of is that the link between SW-1 and SW-2 is broken.
Our luck is that we have CDP enabled and we can check if both switches are seeing each other as CDP neighbours:
We now have the confirmation that the link is working fine, as you can see in the above outputs.
Remember that the hosts are in VLAN 10, therefore the interfaces on both switches should be either in access mode in VLAN 10 or they should allow VLAN 10 on the interface if they are trunk interfaces.
Let’s check the interface GigabitEthernet 8/3 from SW-1:
So the interface is a trunk interface, allowing ALL vlans.
By comparing the “Trunking VLANs Enabled” section of the two outputs, we can conclude that SW-2 doesn’t allow VLAN 10 on the trunk interface towards SW-1.
We have to allow VLAN 10 and see if the MAC address of Host-1 will be present in SW-2 MAC address table:
Now SW-2 can send traffic to Host-1 as it has the MAC address to which it should send the traffic.
SW-2 should have the MAC address of Host-2 in its MAC address table. Let’s check if we have the MAC address of Host-2, which is connected on interface gi1/0/23:
The first MAC address is the MAC address of the interface gi8/3 from SW-1.
The second MAC address is the MAC address of Host-1.
Because we don’t have any MAC address learned on this interface, the first step would be to check the status of the interface gi1/0/23:
This output explains clearly why there is no MAC address learned:
“GigabitEthernet1/0/23 is down, line protocol is down (err-disabled)”
The interface is down because an error occurred on the port. The port was brought down automatically.
This is how you can check why the port was err-disabled:
Under Reason you can see “psecure-violation.” This means that a port-security configuration is in place at that port and the error condition was met.
Let’s use “show port-security interface” command to check the specific configuration of port-security on gi1/0/23 interface:
How do you read this output?
Port security is activated on this port. In case there is a violation of the expected behavior, then shutdown the port. The maximum number of MACs which is allowed on the port is 1. The last MAC address seen on the port and the one that activated the port security is 8071.1fcf.fc8e. The VLAN over which the MAC was learned is 10.
You can recover from err-disabled state of the port by using shutdown/no shutdown on the interface. But if the condition error is still present, then the port will be err-disabled again.
In this specific case, I intentionally added three devices connected to port gi1/0/23 to trigger the condition error.
In the logs, you can see that almost at the same time, two additional MACs trigger to have gi1/0/23 disabled:
May 30 08:35:29.154 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 8071.1fcf.fc8d on port GigabitEthernet1/0/23.
May 30 09:39:29.794 CET: %PORT_SECURITY-2-PSECURE_VIOLATION: Security violation occurred, caused by MAC address 8071.1fcf.fc8e on port GigabitEthernet1/0/23.
May 30 09:39:30.793 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/23, changed state to down
May 30 09:39:31.799 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/23, changed state to down
Let’s check the configuration of gi1/0/23 for confirmation:
As you know, by default the violation mode is shutdown and the maximum number of MACs allowed on the port is 1. This is why you don’t see them explicitly in the configuration.
After shutdown/no shutdown on the interface, we can see that SW-2 now has both Host-1 and Host-2 MAC addresses:
Let’s check also on SW-1:
Everything seems to be in order, so it’s time to check again the connectivity between Host-1 and Host-2:
Now that we are done with the communication between Host-1 and Host-2, it’s time to check the communication between Host-3 and Host-4:
The two links between SW-2 and SW-3(gi1/0/9 and gi1/0/10) are part of a logical bundle, called Portchannel. The Etherchannel protocol is LACP.
Let’s quickly, as before, check if the MAC addresses of the two hosts are present on SW-2 and SW-3:
And for SW-3:
So we don’t have any port-security problems on the access ports.
Now that we know both MAC addresses of the hosts, let’s see if the MAC of Host-4 is present in the MAC table of SW-2 in VLAN 20:
The MAC addresses should have been learned over interface Portchannel1, but it was not.
Checking the operational status of the interface Portchannel1 on SW-2 we see that the physical links are not in the bundle even though they are configured so:
While this command tells us the status of the Portchannel interface, it doesn’t give us too much information, so we can use the “show etherchannel port” command for detailed information about the physical interfaces:
As you can see, both gi1/0/9 and gi1/0/10 are configured in similar way.
One thing that you should always remember is the mode of the protocol.
For LACP you have active and passive. If both sides are active, then the Portchannel will be established. If only one side is active, then the Portchannel will be established. If both sides are passive, then the Portchannel will not be established.
In our case, SW-2 is configured as passive. This means that we must have the active mode on SW-3.
Let’s use the same command to check the mode:
Busted! SW-3 is configured as passive as well. No wonder the Portchannel didn’t come up.
We will configure SW-2 as active to see if the Portchannel interface will come up:
It’s time to check one more time the connectivity between Host-3 and Host-4:
So we have connectivity between Host-3 and Host-4.
Right now the path of the traffic between Host-3 and Host-4 is SW-2 – SW-3.
The topology is built in such way that there is redundancy in case the link between any two switches is failing. The traffic should be rerouted trough the alternate path.
For instance, the traffic between Host-3 and Host-4 should take the path SW-2 – SW-1 – SW-3 if the Portchannel interface between SW-2 and SW-3 is down.
Let’s simulate the Portchannel1 interface failure on SW-2 and let’s check one more time the connectivity between Host-3 and Host-4:
Well, this is not good. We don’t have redundancy.
As a side note, the redundancy in this case is accomplished by spanning tree protocol. The range of problems that one can have with this protocol is very wide and many times advanced troubleshooting skills are needed. For this article, spanning tree was configured correctly and any issue that might appear is not related to it.
Coming back to our problem, it seems that there is an issue on the link between SW-1 and SW-3.
Nothing is obviously wrong with the configuration of the interface gi1/0/11 from SW-3.
Let’s check the interface from SW-1:
The problem might not be that obvious, but this is a dynamic trunking protocol problem.
Both interfaces are configured “dynamic auto,” hence a trunk link between SW-1 and SW-3 will not be formed. This can be verified by looking at “Operational Mode,” which is “static access.”
If one side is configured as “dynamic auto,” then the other one should be configured as either “on” or “desirable.”
By configuring SW-3 as trunk, we should see the interface on SW-1 operating as trunk as well:
But it still isn’t:
But while I was configuring the interface on SW-3, this message was logged on SW-1:
May 30 23:30:21.658 CET: %DTP-SP-5-DOMAINMISMATCH: Unable to perform trunk negotiation on port Gi8/11 because of VTP domain mismatch
This means that the VTP domains are different on SW-1 and SW-3. If this is happening, then, as the message is saying, the trunk negotiation is not happening.
At this moment you have two choices: either change the VTP domains to be identical on both switches or force the links to become trunks.
We will go and change the VTP domain on SW-3, but first let’s check it and change it afterwards:
The connectivity between Host-3 and Host-4 should have been restored:
Now we have a fully functional and redundant network.
If you reached this point of the article, you have encountered the most common problems that one will see when is operating Cisco switches.
Always in switched networks start troubleshooting from the closest point to the user complaining about connectivity problems.
As a summary you should follow these steps:
- Check if you are learning the MAC addresses of the hosts on the closest switch.
- Check on the next switch if the MAC addresses of the hosts were learned.
- Continue to check all the switches on the path between the hosts.
- Make sure that both source/destination MACs are learned on all switches along the path.
- If the MAC is not learned by some switches along the path, then check if the correct VLANs are configured and if they are allowed on the trunk interface.
- If Etherchannel is used, make sure that you use the same protocol on both sides, LACP or PAGP.
- Make sure that you configure at least one side of the Etherchannel as active or desirable.