Many people shy away from enabling debugging of any kind on a Cisco router because they often don’t understand the process well enough. This article will provide an objective explanation of debugging; listing the risks and how to minimize them, as well as review the most commonly used debugging commands.
A common misconception about debugs is that in order to use them, you have to completely understand them. Fortunately, getting something useful from the majority of debugs available on a Cisco router isn’t rocket science. Perhaps the best explanation about debugs is to think of them as an intern poring over the blueprints for a rocket in an attempt to see if he can find anything that looks slightly abnormal.
With constructive internet research, it is usually not difficult to get yourself pointed in the right direction.
Debugs give you a look “under the hood” as to what the router is doing. Rather than waiting and hoping the router finally shows up, or the tester on the conference bridge tells you it’s working, a debug can allow you to see everything happening on the router. A lot of engineers execute a change on a device, and then have to speculate about what the router is doing. Debugging often shows the programmed logic of the router at work as it acts on incoming information, which gives the user a better idea as to why something is happening and how to make something specific occur.
Since the level of debugging is in such great detail, it is of key importance to make sure you are enabling the right debugs so you get the information you need. Turning on too many debugs makes it impossible to go through the information. For both performance and ease of interpretation, you want to be as specific as possible when you turn on debugs.
Before we get into turning on debugs, we should look first at how to minimize any potential impact that turning on a debug might have, because it can have unintended consequences if you are not careful.
Capturing Debugging Output
With debugs, you obviously need a place to store the information generated by the debug. Where this is depends on where the router is configured to print the debug output. They include the console port (if you are connected to the router via the console), the VTY/AUX lines, to a syslog server, and to the buffer of the router. When choosing where you want the debug output to go, you should remember a couple things. First, is where it is being sent going to prevent you from turning it off? For example, if you are connected via a console port and you enable a debug that prints every frame passing through the router, it is likely going to be very difficult for you to turn off that debug, and you risk potentially affecting production traffic, which is never a good thing. Secondly, you want to ask yourself if where you are sending the debug output going to allow you to review it later. A scrolling debug is great, but if your logging buffer wraps after 200 lines, you will not be able to get a comprehensive view of what is going on, especially given the amount of output some debugs are capable of generating. As a general rule, the best output location will be one of two places – either in the logging buffer of the router (visible with the “Show log” command), or sending all the output to a syslog server. If you choose the logging buffer option, you want to make sure that the logging buffer level is set to debugging. By default, a Cisco router will only send messages generated by the router of a certain severity to the logging buffer.
Also, you want to make sure your buffer is big enough. There is a single command that can set both of these options: logging buffered <size> debugging.
If you send the output to a syslog server, you must use the “logging trap debugging” command. Otherwise, only informational severity messages get sent to the syslog. Again, you should be aware that it requires CPU cycles for the router to package all the output into IP frames and get it off to the syslog. Be wary if you are enabling debugs with exceptionally large amounts of data because it takes bandwidth to send all that data. If your site is already running at 95 percent utilization, a 500Kbps debug stream can push it over the edge.
With all this being said, enabling a debug is not difficult. As you might guess, switching on a debug occurs with the word “debug” on the router. All this is done from privileged mode – you don’t want to be in config mode when you do this. Experienced engineers have memorized a few debug commands, but the question-mark is your friend! The hierarchical nature of the Cisco configuration commands is closely mirrored in the debugs. For example, you might debug why a static route (“ip route” command) isn’t getting installed with the “debug ip routing” command. Find a command you think will work and before you use a command, research Cisco’s website to see if there are warnings about that command causing extreme CPU load, etc. Cisco puts out an entire debug command reference, which you should be able to find for the version of IOS you are running.
Limiting Debug Output
When enabling debugs, Cisco has given us a couple good ways to limit the impact and amount of data generated by what would otherwise be significantly taxing commands. These options are not available on most debugs, but are there for some of the most commonly used ones. Perhaps the two main ones are access-lists and conditions. Access-lists are straightforward. You define the specific component of the larger debug you are interested in and the router displays only information for the debug for that specific component. For example, if you are trying to see how a router is routing packets to a specific destination, you would first define the prefix you are interested in. For example, destination 18.104.22.168 in an access list. Then, you match that access list in the debug command. It would look something like this:
access-list 1 permit 22.214.171.124 debug ip packet 1
The other common way to limit debug output is a condition. This is commonly used for interfaces and with IPsec. For example, if you wanted to enable a broad debug for a specific IPsec/crypto peer, you would enable a debug crypto condition to match that peer first, than enable the broad debug. Rather than applying to all IPsec peers, that broad debug applies only to that specific peer.
Use caution when removing a condition if you leave the original broad debug in place! You can see what debug conditions are running with the “show debug condition” command.
For an interface, you can apply a condition that only enables the debug for traffic transiting the interface whose condition you specify.
Depending on how worried you are about losing access to the device when running a CPU-intensive debug, you can set a timed reload, or perhaps open another session, and have the undebug all output typed so all you have to do is hit enter on the second terminal session to stop the output. It’s nice to know you’ve got some options here.
Another thing with debugs to keep in mind is that in general, a debug is only going to show information about data/decisions that are processed by the CPU of the router. If something is being switched in hardware, it may be difficult to see. This is why it’s generally a lot harder to use debugs effectively on a switch. Switches will often use a SPAN session to mirror traffic passing through the switch to another port for further analysis with.
Commonly Used Debugging Commands
Now that you have an understanding on why debugs are useful, how to log the data from them, as well as a few of their basic features and caveats, we will take a look at some of the most commonly used debugs. As always, the best debug is going to depend on what issue you’re seeing and what you’re trying to find out.
One commonly used debug is the “Debug ip routing” command. This debug will show you why a router is changing the routing table. Don’t get confused here and think this command will show you every packet routed through the router. That’s a different command we’ll cover later. “Debug ip routing” shows why the router is changing the routing table. For example, if you are learning a certain destination prefix (126.96.36.199/32) via both OSPF and EIGRP and the OSPF route is taking precedence, you want to know why. The “debug ip routing” command is going to show you exactly when and why the routing table is being changed for 188.8.131.52/32.
Another great debug is the “debug ip packet” command, which will show you every packet passing through the router. Again, for anything to show up in a debug, it must be processed by the CPU. If you have a mechanism like route-cache or CEF turned on for the input interface of the traffic you are interested in, you are not going to see it. You will need to disable that temporarily. I should note here that you want to do this with some deference. In general, it’s not going to cause an issue, but it is something to be aware of. Using this command is a great time to use one of the conditions we spoke about earlier to limit the impact. If you don’t, the router could potentially try to print every single packet of output to the location you’ve specified it to go. Unless there isn’t a lot of traffic, this will cause some serious problems.
Another good couple debugs are “debug crypto ipsec” and “debug crypto isakmp.” Usually they mean VPN tunnels, but anything you configure with a crypto command in running-configuration is generally going to be picked up here. Be aware that ISAKMP includes Phase 1 negotiation and IPSEC includes Phase 2 negotiation. If you have a lot of IPsec tunnels, a crypto condition matching the specific peer you are interested in prior to enabling the isakmp/ipsec debugs is something you will want to consider.
Another good debug is “debug ip bgp.” There is a host of sub-options beyond debug ip bgp, but in general this will show information about BGP routing updates being sent/received for a given prefix. As BGP is the most widely used with the most complex metric of any routing protocol you can configure on a router, this can be an invaluable troubleshooting tool when you’re wondering why a certain route isn’t showing up when you know it should.
One other good thing to remember when you’re searching for the perfect debug command is that Cisco likes to hide some of the really great ones under the “debug ip” subcontainer. OSPF is one such example. If you do a debug os and put a question mark at the end, you will not see anything. Cisco puts the OSPF debugs under “Debug ip ospf” container. I can’t stress how much you need to use the question mark when first using the Cisco debug capability. There is simply no way you can remember them all.
After you have enabled a debug, one of the first things you want to check is whether the CPU of the router has changed significantly in the past few seconds. If it is doing OK, you can rest assured everything is going to be fine.
The debug spanning-tree commands are very helpful when troubleshooting instability relating to spanning-tree. I should note that exceptional care and caution should be used when enabling these debugs, as it is easy to lock yourself out of the router, simply because of the amount of output this command is capable of generating. For example, if you have 300 VLANs configured, each of which is receiving and/or sending a BPDU every two seconds, this is a tremendous amount of output. This is another example of a time when you want to use a debug condition to limit the output – such as “debug condition vlan xx” for the specific VLAN you are interested in. Debug spanning-tree bpdu shows the information the spanning-tree algorithm uses to decide what ports will be root versus designated versus blocking, the spanning-tree root selection, and a host of other data.
In summary, debugging (when used well) can be a network administrator’s best friend. It can tell you why a router does what it does, and will get you pointed in the right direction when you’re trying to solve a problem.