Many people shy away from enabling debugging of any kind on a Cisco router. As usual, the reason for shying away from something is lack of understanding. This article attempts to provide an objective look at debugging, listing the risks and how to minimize them, as well as some commonly used debug commands.
A common misconception about debugs is that in order to use them, you have to completely understand them. Fortunately, getting something useful from the majority of debugs available on a Cisco router isn’t rocket science. Perhaps the best explanation I have heard about debugs recommended thinking of them as an intern poring over the blueprints for the rocket and trying to see if he can find anything that looks slightly abnormal.
Then, with some constructive Googling, it is usually not all that difficult to get yourself pointed in the right direction.
Debugs really give you a look “under the hood” as to what the router is doing. Rather than waiting and hoping that the route finally shows up, or the tester on the conference bridge tells you it’s working now, a debug can allow you to see everything.
A lot of engineers execute a change on a device, and then wait and wonder what the router is doing. Debugging often shows the logic and coding of the router at work as it takes incoming information and performs certain actions based on that. You can get a much better idea as to why something is happening and how to get what you want to happen occur.
Since the level of debugging is in such great detail, of key importance is to make sure you are enabling the right debugs so you get the information you need. Turning on too many debugs makes it impossible to go through the information. You want to be as specific as possible when you turn on debugs, for both performance and ease of interpretation purposes.
Before we get into turning on debugs, we should first look at how to minimize any potential impact turning on a debug might have – because it can have some unintended consequences if you are not careful.
With debugs, you obviously need a place to see the information generated by the debug. Where this is depends on where the router is configured to print the debug output. They include the console port (if you are connected to the router via the console), the VTY/AUX lines, to a syslog server, and to the buffer of the router.
When choosing where you want the debug output to go, you should remember a couple things. Is to the destination going to prevent you from turning it off? For example, if you’re connected via a console port, and you enable a debug that prints every frame passing through the router, it’s likely going to be very difficult for you to turn off that debug. You will also leave yourself wide open to potentially affecting production traffic – and that is never a good thing.
Secondly, is the destination going to allow you to review it later? A scrolling debug is great, but if your logging buffer wraps after 200 lines, you’re likely not going to be able to get a good idea of what is going on, especially given the amount of output some debugs are capable of generating.
As a general rule, the debug output will go to one of two places: either the logging buffer of the router (visible with the “Show log” command) or a syslog server.
If you choose the logging buffer option, you want to make sure of two things. First, the logging buffer level should be set to debugging. By default, a Cisco router will only send messages of a certain severity generated by the router to the logging buffer. Also, you want to make sure your buffer is big enough. There is a single command that can set both of these options – Logging buffered <size> debugging. The default severity sent to the syslog server is XX.
If you are going to send the output to a syslog server, you must use the “logging trap debugging” command. Otherwise, only XX severity messages get sent to the syslog. Again, you should be aware here that it requires CPU cycles for the router to package up all the output into IP frames and get it off to the syslog. Be wary if you are enabling debugs with exceptionally large amounts of data.
With all this being said, enabling a debug is actually not all that difficult. As you might guess, switching on a debug occurs with the word “debug” on the router. All this is done from privileged mode – you don’t want to be in config mode when you do this. The honest truth is that highly experienced engineers have a few debug commands they have memorized, but remember: the question-mark is your friend!
The hierarchical nature of the Cisco configuration commands is fairly closely mirrored in the debugs. For example, you might debug why a static route (e.g. “ip route” command) isn’t getting installed with the “debug ip routing” command. Find a command you think will work and maybe the first time you use it, do some strategic Googling to see if there are big warnings on Cisco’s website about that causing extreme CPU load, for example. Cisco even puts out an entire debug command reference which you should be able to find for the version of IOS you are running.
When enabling debugs, Cisco has fortunately given us a couple of good ways to limit the impact and amount of data generated by what would otherwise be significantly taxing commands. These options are not available on most debugs, but are there for some of the most commonly used ones. Perhaps the two main ones are access-lists and conditions.
Access-lists are pretty straightforward: you define the specific component of the larger debug you are interested in, and the router displays only information for the debug for that specific component. For example, if you are trying to see how a router is routing packets to a specific destination, you would first define the prefix you are interested in – let’s say destination 18.104.22.168 – in an access-list. Then, you match that access-list in the debug command. It would look something like this:
access-list 1 permit 22.214.171.124
debug ip packet 1
The other very common way to limit debug output that Cisco has provided to us is a condition. This is commonly used for interfaces and with IPsec. For example, if you wanted to enable a broad debug for a specific IPsec/crypto peer, you would enable a debug crypto condition to match that peer first, and then enable the broad debug. Rather than applying to all IPsec peers, that broad debug then applies only to that specific peer.
As you might guess, you’re going to want to use a fair amount of caution when removing a condition if you leave the original broad debug in place! You can see what debug conditions are running with the “show debug condition” command. For an interface, you can apply a condition that only enables the debug for traffic transiting the interface whose condition you specify.
Depending on how worried you are about losing access to a device when running a CPU-intensive debug, you can set a timed reload, or perhaps open another session, and have the ‘undebug all output’ command typed so all you have to do is hit enter on the second terminal session to stop things. It’s nice to know you’ve got some options here!
Another thing to keep in mind with debugs is that in general, a debug is only going to show information about data/decisions that hit the CPU of the router. If something is truly being switched in hardware, it may be difficult to see it. This is why it’s generally a lot harder to use debugs effectively on a switch. Switches will more often use a SPAN session to mirror traffic passing through the switch to another port, for further analysis with a tool like Wireshark. But, that is a different discussion altogether.
So now that you have a solid handle on why debugs are useful, how to log the data from them, as well as a few of their basic features and caveats, we can look at some of the most commonly used debugs. As always, the best debug is going to depend on what issue you’re seeing and what you’re trying to find out.
One commonly used debug is the “debug ip routing” command. This debug is going to show you why a router is changing the routing table. Don’t get confused here and think that this command will show you every packet routed through the router. That’s a different command we’ll cover later.
Debug ip routing shows why the router is changing the routing table. For example, let’s say you are learning a certain destination prefix (126.96.36.199/32) via both OSPF and EIGRP. For some reason, the OSPF route is taking precedence, and you want to know why. The debug ip routing command is just the ticket. It’s going to show you exactly when and why the routing table is being changed for 188.8.131.52/32.
Another great debug is the debug ip packet command. This is the command that will show you every packet passing through the router. One caveat here is that for anything to show up in a debug, it’s got to have to hit the CPU! So, if you have a mechanism like route-cache or CEF turned on for the input interface of the traffic you are interested in, you are not going to see it. You will need to disable that temporarily.
I should note here that you want to do this with some deference. In general, it’s not going to cause an issue but it is something to be aware of. This command is a great time to use one of the conditions we spoke about earlier to limit the impact. Otherwise, you have the potential that the router is going to try to print every single packet of output to the location you’ve specified it to go. And unless there isn’t a lot of traffic at all, this will cause some serious problems!
Another couple of good debugs are “debug crypto ipsec” and “debug crypto isakmp”. Both are related to – you guessed it – crypto! Usually this means VPN tunnels, but anything you configure with a crypto command in the running-configuration is generally going to be picked up here. ISAKMP includes Phase 1 negotiation and IPSEC includes Phase 2 negotiation – so be aware of this. Again, if you have a ton of IPsec tunnels, a crypto condition matching the specific peer you are interested in prior to enable the isakmp/ipsec debugs is something you will want to consider.
Another good debug is “debug ip bgp”. There are a whole host of sub-options beyond debug ip bgp – but in general this will show information about BGP routing updates being sent/received for a given prefix. As BGP is the most widely used and easily has the most complex metric of any routing protocol you can configure on a router, this can be an invaluable troubleshooting tool after you’ve banged your head up against why a certain route isn’t showing up, when you know it should.
One other good thing to remember when you’re searching for the perfect debug command: Cisco likes to hide some of the really great ones under the “debug ip” subcontainer. OSPF is one such example. If you do a “debug os” and don’t see anything, you will find it under the “debug ip ospf” container. Again, I can’t really stress how much you need to use the question mark when first using the Cisco debug capability. There is simply no way you can remember them all.
After you have enabled a debug, one of the first things you might want to check is whether or not the CPU of the router has changed significantly in the past few seconds. If it is doing OK, you can probably be pretty well assured that everything is going to be just fine.
The debug spanning-tree commands are very helpful when troubleshooting instability relating to spanning-tree. I should note that exceptional care and caution should be used when enabling these debugs, as it is pretty easy to lock yourself out of the router since it prints out so much debug output.
For example, if you have 300 VLANs configured, each of which is receiving and/or sending a BPDU every 2 seconds, this is a tremendous amount of output. This is another classic example of a time when you’d want to use a debug condition to limit the output such as “debug condition vlan xx” for the specific VLAN you are interested in. Debug spanning-tree bpdu shows the information that the spanning-tree algorithm uses to decide what ports will be root versus designated versus blocking, the spanning-tree root selection, and a whole host of other data.