Very few people understand the complex mechanisms used to send and receive data over a network. And from that number, perhaps even fewer are able to decipher anything relevant from a troubleshooting perspective by looking at the actual bits and bytes that are flowing over a network. Fortunately, there are some tools that, when used appropriately, can provide in depth, relatively user-friendly traffic analysis.
Perhaps the most common tool used to do this is an application called Wireshark. Wireshark is a free, open source application that can be downloaded and installed by anyone. Wireshark works by capturing traffic on a NIC (network interface card). The NIC it captures data from is usually going to be a wired Ethernet adapter or a wireless adapter. It can then analyze individual frames, a unique conversation between two hosts, or a number of other possibilities to provide the user with a tremendous amount of information about what’s going on in their network.
Wireshark isn’t unique in its ability to capture traffic.There are many utilities that can do that. TCPDump, for example, is built into many Linux and UNIX operating systems. Where Wireshark truly excels is in its ability to analyze the traffic that is captured. The captured data also doesn’t even need to be gathered by Wireshark. A Linux admin can use TCPDump to gather some traffic, and then load that file into Wireshark for further analysis.
In this write-up, we’ll be looking at some of the basics regarding how to use Wireshark, a basic understanding of how TCP works, and some basic things you can check within Wireshark itself.
Of course, the first thing you do is install the application. If you are installing on a Windows machine, the Wireshark package also includes a program called WinPcap. WinPcap is a program that enables your network adapter to listen for traffic, even if that traffic isn’t actually destined for your machine. Once installed, you want to start the capture. To start a capture, you click on the start capture button.
When you click this, you will see a list of all the NICs on your system that Wireshark can capture from. If the names of the interfaces aren’t enough of a hint, Wireshark also lists how many packets are being sent or received on each interface. Usually, the one that’s seeing all the traffic is the one that you are going to be interested in.
If you are trying to capture traffic from your wireless adapter, you may or may not have issues. Some wireless NICs will show a ton of wireless medium management (RTS, CTS, etc.,) frames, while others will show the traffic on a non-wireless interface, or perhaps a combination of all of the above. Really, what you’re looking for is the interface with the greatest amount of traffic.
Once you’ve started the capture, your screen will likely be flooded with all sorts of packets and frames you never knew existed. And honestly, as long as your performance is good, this is not necessarily a problem. But, if you’re experiencing things like slow file transfers, FTP sessions that reset intermittently, or an infinite number of other possibilities, Wireshark is the “go to” tool for getting under the hood of what is going on in the network.
The first thing you should do when using Wireshark to troubleshoot a problem is to get some details from the user reporting the issue. This may sound like a trivial non-technical recommendation, but it is absolutely critical if you want to do anything other than waste time playing around in Wireshark. For example, is the problem slowness, or that the session resets? Do all applications experience the problem, or just one? Does it occur at a specific time of the day? How long does the issue persist? When did it start happening?
Finding out the answers to these questions will help you to avoid going down the proverbial rabbit hole. And believe me, Wireshark provides plenty of just such rabbit holes. It is so, so easy to see something that is not a problem as the culprit while glazing over the real cause. A good and firm problem identification doesn’t guarantee this won’t still happen, but certainly makes it less likely.
Also, when getting packet captures, make sure you get one of when the problem is happening as well as when the problem isn’t happening. Finding what is normal versus what is not normal can also prevent a false positive.
Finally, the known good and known bad captures should be sufficiently long to really get a good idea of what’s going on. Two 10-second captures are very vulnerable to having some anomaly in them that can falsely be identified as either normal or abnormal, when that isn’t really the case.
With those generic points out of the way, we can now get into some more specific tools within the application itself.
In most cases, transferring data between two users occurs via an application-level protocol running over TCP. Without going into too much detail, you should know that TCP infers reliability in delivery. This just means that after a sender has sent a section of data to a receiver, it won’t send any more data until the receiver indicates that it received the last section of data. If you are thinking that this sounds inefficient, you are right in many ways. But, without it, a sender would send data at a rate that might be wrong for the receiver it is communicating with, not to mention the possibility that frames can get corrupted, lost, or delayed in transit. Without TCP facilitating communication between the sender and the receiver, it would be very difficult to transfer data over the network.
The very first thing that is done at the beginning of every TCP-based data transfer is a process called a 3-way handshake. A 3-way handshake is just a series of 3 packets exchanged between the sender and receiver that gets both sides synced up and ready for the transfer to begin.
The first part of a 3-way handshake is the SYN packet. A receiver sends a SYN packet to a sender on a port the sender is listening on. When the sender receives that SYN packet, it replies with a SYN/ACK packet. It’s basically the sender’s way of accepting the receiver’s request to connect. The 3-way handshake completes when the receiver sends a final ACK packet back to the sender. When this is complete, both sides are ready to send and/or receive data.
When a user complains that a website is “hanging” or “not loading” – there is a very good possibility that the 3-way handshake is not completing successfully. In this case, you’d see a series of SYN packets, but never a SYN/ACK packet coming back. Usually these SYN packets are sent by the receiver every 15-30 seconds. Eventually a client will just give up and quit sending them. This is usually the same time you’d get a message in your browser about the page not being able to be displayed.
So let’s assume that the session always seems to start fine but the user is still complaining about something else (slowness is a common one).
If you have multiple TCP-based conversations going on at the same time – which is virtually inevitable for the average user – Wireshark has a very nice feature whereby you can show only packets from a given TCP session. It’s as simple as finding one packet from the conversation you’re interested in, right clicking on that packet, and choosing the “Follow TCP Stream” option in the resulting menu that pops up. Wireshark automatically determines which packets are in that TCP stream and displays only those.
Considering that a capture of only a few minutes can contain literally hundreds of thousands of frames from different TCP sessions all mixed in together, this makes analysis infinitely easier. At the top of this TCP stream in the Wireshark window, you should see the SYN, SYN/ACK, ACK sequence of the 3-way handshake that started the conversation.
With a TCP-based transfer, we already discussed that the delivery of data is reliable. The way that both sides of a conversation keep track of what data is sent and received is through special sequence numbers and acknowledgment numbers located in the TCP header. Each 1 byte of data payload transferred between hosts correlates to an increase in the sequence or an increment of the ACK number by 1.
In the example below, you see the 3-way handshake. The first packet after the 3-way handshake sent by the client to the server has a sequence number and acknowledgment number of 1. The length of 194 indicates that my PC is sending 194 bytes of payload (payload is the data that’s left after you remove all the extra headers – IP, TCP, Ethernet, etc.).
The server then responds with an acknowledgment frame. The ACK number in this frame is 195. This means that the other side received 194 bytes of data from me, and the next sequence number it’s expecting from me is 195.
Having now received confirmation of successful delivery, my machine is able to send the next bit of data.
The reason I go into detail on this process is because it is critical to understand much of what Wireshark is going to be able to tell you about a conversation. For example, Wireshark can show you a graph of the increase in the sequence numbers over time, the bytes transferred over time, or the speed of the transfer. You can view these graphs in a couple different formats – Stevens or TCP trace. While I prefer the Stevens format, it is completely up to you. You can create and view this graph by going to Statistics -> TCP Stream Graph and selecting from the two.
One caveat here is that you want to select a packet whose source IP address is the one transmitting the data. Since data flowing both ways is tracked by TCP, you must do this to ensure you aren’t just analyzing the other side’s acknowledgements to the data you’re sending.
A healthy file transfer should look fairly smooth, with no major plateaus. Remember what a plateau means – it means that sequence numbers were not increasing as time increased – meaning data wasn’t being transferred at all – or at least as fast – during the plateau.
With a good understanding of how TCP works to ensure reliable delivery, you should also understand what TCP does when it doesn’t get a packet. Let’s say your transfer is progressing smoothly, and all of a sudden the host receiving the data sees a gap from the sequence number it last acknowledged to what it’s getting now. In basic terms, this means that some of the data sent to the receiver got lost along the way.
The receiver immediately informs the remote side that it did not receive some of the data sent by the sender, and that in order for TCP to continue, the sender needs to retransmit it. The way the receiver does this is through what is referred to as a duplicate acknowledgment (or Dup ACK). This is just an ACK with the sequence number of the packet the host didn’t receive. This is a signal to the server that it needs to resend that packet. It’s quite common for a receiver to send MANY duplicate ACKs before the server finally retransmits the requested data.
Even though retransmissions by the sender are often required, another very common occurrence is that one packet just got delayed in its arrival. In this case, you’d still see the duplicate acknowledgments from the receiver, but they would shortly be followed by the “missing” packet. Wireshark kindly flags this packet as being out of order. Once the missing packet is received, the transfer can continue.
Whether the solution is a full retransmission or just waiting a few microseconds for an out of order packet, you should realize that when this happens you are slowing down the rate of transfer.
When looking at a Wireshark trace, the occasional out of order packet and retransmission is inevitable. What you are usually looking for are very long, repetitive strings of them that are in stark contrast to the baseline “known good” capture you took earlier.
Fortunately, rather than parse through the hundreds upon hundreds of thousands of packets in a TCP stream, Wireshark has provided a couple of ways to identify trouble frames very quickly.
One way is to go to Analyze -> Expert Info. In the resulting pop-up box, there are several tabs with issues grouped into each tab according to how serious Wireshark deems that issue to be. Within each tab, each issue lists the number of the frames that had that issue (Count). And, if you expand that issue, and click on the displayed packet numbers, Wireshark takes you exactly to that frame in the capture!
So let’s say as you are becoming more familiar with the issue, you are interested in seeing all the frames that meet specific criteria – or even set of criteria. For example, you wanted to see all TCP retransmission frames, or wanted to see frames that are above a certain size. Wireshark has a virtually endless range of filters you can put in place that will limit the frames displayed to those matching or not matching the specific criteria.
All this is done in the filter box towards the top of Wireshark.
While you can read up on all the possible filters and try to memorize them, it’s usually much easier to drill down into the field you are interested in, right click, and let Wireshark create it for you. You should note here that not only can you use match logic, you can also use match not logic – as well as matching or not matching certain multiple things together at the same time. Once you’ve defined the perfect filter you’re looking for, you can review the actual filter “code” at the top of the screen that Wireshark built for you, and you can save it for future use or reference if you so desire.
Like most topics in networking, this is not meant to be a comprehensive guide of everything you can do in Wireshark. Indeed, it barely scratches the surface. In any case, these tools provide a good start for an initial analysis.