Many times, by the time you realize there's a problem, it's in the past. We had a situation where a couple of floors of the network would "go offline" for ~10 minutes at random times every few days. Reports would come in, sometimes hours, after the fact when the network was back to normal.
Wireshark and its command line companion (tshark) have the ability to create circular/ring buffers. In effect you can capture the "last x hours" of data. As it's capturing data, it will overwrite the oldest data, keeping the most recent. We configured a raspberry pi with 2 ethernet ports - one for management, one for the span destination. It had a largish ssd drive - enough to capture ~1 days traffic. We placed it at the client site, set up a circular capture buffer and span on the switch and let it run. When we received word that we had an incident we remotely connected to the monitoring machine stopped the capture and retrieved the buffers. In this case, with Linux on the pi, we actually used tcpdump which also supports circular buffers and whose files are natively readable by wireshark. Some additional tips:
- Ring buffers composed of many small buffers are typically better than a small number of very large buffers. Once you know the time frame, you can pick the relevant buffer(s) and more easily search/analyze/sort small buffers quickly rather than large buffers which can take exceedingly long to do anything. Hundreds of 200M buffers worked well in our case.
- Make sure you configure the span to capture incoming and outgoing packets.
- If you're analyzing encrypted conversations, you may want to investigate capturing headers only since the payload is unreadable (this may be the tcpdump default). This should reduce the capture size. I have not tried this so its usefulness is unknown.
- TCPdump is fiendishly useful and integrates well with wireshark. It's good to learn some basics if you use linux (you should...) Google is your friend.
- We have a number of Cat 9K switches onsite we could use to capture packets onto the switch - I believe you can also configure ring buffers there. Unfortunately, there are severe restrictions on capture sizes, syntax differs between different models and scant online docs always seem to reference that other syntax... It's maddening. I would hope you could output to a large USB stick to remove size restrictions but that appears to not be a usable option for this situation. However, since *you* know both endpoints of your conversations, this may be a useable option for you (assuming you have cat 9Ks).
For those interested, our issue turned out to be some Dell machines with a bad network drivers that would start pumping out IPv6 traffic (non stop and lots of it - effectively DOS'ing the local network). Updating the drivers fixed the issues -its a known bug. We've seen this issue now multiple times. Each time we're forced to shut down the access ports until the desktop support people update the client's drivers.
Cheers,
Paul
Lead Network Engineer, University of Illinois at Chicago
