Methods of capturing packets on CUBE during high call volume ?

[joelalli123]

I have an ASR1k (CUBE) that is pushing about 100Mb continuously. The type of traffic is SIP/RTP. I need to capture this traffic and buffer it for about 4 hours for troubleshooting purposes.

The issue is that the SIP provider is randomly dropping the RTP stream during calls and unfortunately there are no SIP signaling errors. It turns into a bunch of finger pointing by all parties involved. 

I know there are commands that could be used to diagnose the issue while the call is connected however, I do not receive the bad call examples until long after the problem occurred. Also, I know there are debugs that could help but that will push the CPU too hard.

I have yet to find a graceful solution. I know there are IP Traffic Export and Embedded Capture features on IOS but they really weren't designed for permanent configuration.

I was considering Wireshark on a SPAN port but then the issue is, can Wireshark even handle that much BW? I've never seen Wireshark get even close to that BW without crashing. In addition, assuming I got Wireshark working, there would be issues with isolating a call in the massive Wireshark logs, or in the countless Wireshark files (if set to create small files).

Any recommendations? I would even consider 3rd party monitoring/capturing apps if they did what I need.

[Mohammed al Baqari]

Hi,

One option is to capture using filters such as capturing signaling only instead of the entire traffic pipe. 

If you would like to capture the entire pipe, short answer you can't do it using softwares or using built-in features in IOS/IOS-XE. At such high data rate, the process will crash. You need to have dedicated hardware appliances which can perform capturing at such data rate (e.g. NetShark from riverbed).

[joelalli123]

Thanks Mohammad. I'll check out Riverbed.

[Dennis Mink]

Wireshark is by far your easiest and cheapest option. use span and a machine with a SSD.  also make sure you define capture filters to only grab 5060/5061 and RTP. 

If its a matter of RTP packets getting randonly dropped, start checking your QoS policy maps

also it might be worth running a stress test, to see if other traffic is pushing RTP out when there is congestion.

Please rate if useful

[joelalli123]

Unfortunately it's not an issue with congestion. It usually happens after the RTP has changed from a=Inactive to a=SendRecv -or- at the beginning of the call after the first SDP is negotiated. Since everyone is agreeing on the SDP (no signaling errors) someone is probably not listening to the correct UDP port, or sending to the wrong port. 

Good idea on the machine with SSD. I might try that.

If you know of a way to track calls that are in a SendRecv state but there is no RTP stream (SNMP?), please let me know. Thanks!