Hello and welcome back to this series on Network Reliability. In the last article, we started the series by introducing network reliability and redundancy and we also examined single points of failure and types of network redundancy. You can view the post here.
In this article, we will focus on Layer 2 redundancy. We will begin by examining switched network designs and then move on to active standby redundancy in switches using Spanning Tree Protocol. Finally, we will examine active/active redundancy in layer 2 networks using per vlan spanning tree protocol and Multiple Spanning Tree Protocol.
Just before we dive into redundancy, let’s re-examine a few basic concepts. Multiple devices can be connected together on a network using a switch. These switches, as opposed to hubs, separate the collision domain and as such, they improve connectivity. These switches operate at the data link layer (Layer 2) of the OSI model and they switch frames from one port to the other based on their destination MAC addresses. The MAC addresses and port information are learned dynamically (from the source MAC address of frames ) and stored in a MAC table on the switch. Finally, switches can be further segmented into different broadcast domains using VLANs.
When we have more devices/users than a switch can handle, we usually introduce another switch and in large networks, these switches can easily become hundreds, or even thousands across the switched domain.
Cisco, and other large network vendors, recommends a hierarchical design (tiered design) where lower capacity switches are used to provide access to users and switches with a higher capacity and throughput are deployed at the distribution and core layers. Let’s consider our simple design from the last article:
Active/Standby Redundancy in a layer 2 switched domain
In the diagram, the access switches are connected directly to an aggregation switch which is connected to other services like the server farm and the internet service provider. In this case, the aggregation switch is clearly a single point of failure. (You might argue that the access switches are also single points of failures because all the users connected to them would lose access if they go down. But for this article, let us assume we have cold standby in place for access switches). To improve reliability, the logical decision would be to add another aggregation switch for some redundancy. After a redundant switch is handed and all the access switches are connected to the switch, we would have the network below:
By adding more links and switches, we have made the network more redundant but this does not necessarily translate into reliability. This is because of the way switches operate together on a network.
In order to prevent Layer 2 loops on a switched network, Switches use the Spanning Tree Protocol to determine which ports can forward frames at a particular time. The Spanning Tree Protocol is an algorithm which was developed by Radia Perlman (and standardized as the IEEE 802.1D protocol). Basically switches select a root bridge and based on their closeness to the root bridge, they determine root ports, designated ports and blocking ports. Any port that is determined to be a blocking port would not forward traffic to prevent loops from occurring on the network. When there is a change on the network, the algorithm is run again and ports are assigned new roles. You can read more about the spanning tree protocol here.
So what has spanning tree protocol got to do with switch and link redundancy? Everything! In fact, if spanning-tree is not properly optimized, there would be little or no real benefit of adding the new switches in the network. And these switches can cost tens (or hundreds) of thousands of dollars. Looking at the network above, what could go wrong? If one of the access switches becomes the root bridge of the network (the root bridge is selected by choosing the bridge with the lowest bridge ID, and by default, this is the switch with lowest MAC address), then the blocking ports on the network would be as shown below:
As seen in the diagram above, most of the traffic on the switched network would be forced to go through a switch with low throughput. Inevitably, there would be a lot of congestion on the network. To prevent this, we have to ensure that one of the aggregation switches becomes the root bridge of the network. To force a switch to become the root bridge, we can set the priority of the switch to a lower priority. But what if we don’t know the priority of the other switches? Cisco has a “cheat command” for that. It’s called the “spanning-tree root” command. So we would apply the command to one of the switches.
AGG_Swith1(config)#spanning-tree vlan 1 root primary.
Using a similar syntax, we can make the other switch the secondary root bridge for the network.
AGG_Swith2(config)#spanning-tree vlan 1 root secondary.
With the right root bridges set, the new blocking ports on the switches would be as shown below;
Now, we have traffic flowing in the right direction (higher capacity switches and links are being used) and we also have some form of redundancy on the switches. However, if we closely look at the diagram, we would see that almost half of the inter switch links are still in blocking mode, and this means that unless there is something wrong with the network, these links would never be used. Again, this is not cost effective and in some cases, we might even be overloading the primary links that are carrying traffic. Typically, this is the case with Active/Standby redundancy.
Active/Active Redundancy on a Layer 2 Switched Network
For Active/Active redundancy in an L2 switched network, we have to separate the spanning tree domains. The goal here is to alternate the root bridges so that each core device (the aggregation switch in this case) is the root bridge for a particular instance. That way, different links would be in blocking mode for different instances of the spanning tree protocol. Using this kind of configuration, the network would look like this:
Here we can see that for instance 1, a different sets of ports (shown with red stars) are in blocking mode, while a totally different set of ports are in blocking mode for instance 2. This means that the ports that were blocking in instance 1 are forwarding in instance 2. So, all the ports are being used. However, if one of the aggregation switches fails, then the other aggregation switch becomes the root bridge for both instances and then we would go back to the scenario in the previous diagram.
Okay, that sounds a lot better. But how do we implement these “instances” in our L2 network? One word – VLANS! Yes, with VLANs, we can run multiple instances of STP. In Cisco terms, this is called Per Vlan Spanning Tree Protocol (PVST), and in this variation of STP, each VLAN runs its own instance of spanning tree. Assuming we have two vlans (10 and 20), the configuration on the aggregation switches to support PVST would look like this:
AGG_Swith1(config)#spanning-tree vlan 10 root primary.
AGG_Swith1(config)#spanning-tree vlan 20 root secondary.
And on the second aggregation switch, we would have;
AGG_Swith2(config)#spanning-tree vlan 10 root secondary.
AGG_Swith2(config)#spanning-tree vlan 20 root primary.
PVST is all kinds of cool but it is only a Cisco Proprietary protocol. So what happens when we need active/active redundancy in non-Cisco switches? Or what if we need our Cisco switches to interconnect with other vendors that do not support PVST? Well, thankfully, the IEEE has not left us hanging, there is the Multiple Spanning Tree (IEEE 802.1s,) a variation of the original 802.1D, which can be used to run multiple instances of spanning tree on a switch.
MST ensures that there is backwards compatibility with original STP by using the same BPDUs for all instances of STP running in the region. From a bandwidth perspective, this is also more effective than PVST. You can find more details about MST here.
So in this article, we started taking a deep(er) look at redundancy at the data link later of the OSI model. We examined how the spanning tree protocol affects Layer 2 redundancy. We looked at how a single instance of STP gives Active/standby redundancy in a switched domain. Finally, we looked at how we can achieve Active/Active redundancy using multiple instances of STP either through Cisco’s Per VLAN spanning Tree protocol, or through the IEEE’s Multiple Spanning Tree Protocol (802.1s)
In the next article, we will continue our deep dive by looking at how we can achieve more Active/Active redundancy by bundling using link aggregation and switch stacking technologies.
Thank you so much for reading and I look forward to writing the next article in this series. As usual, feel free to drop your thoughts and questions using the comment box. See you soon!