Output Hold Queue Drops

Mike Assel · ‎01-03-2013

Hello. I am working with a 3560G switch that is dropping packets for the output queue. The only traffic on the switch are UDP mutlicast video streams. The confusing thing was that on the port dropping the packets, the outgoing bit rate was only about 120Mbps (on a 1 Gbps port). After speaking with TAC they advised me that the drops are likley being caused by transient bandwidth spikes. I recieved the slides that are attached that outline a method to detect these spikes using wireshark. The slides make sense to me, but the main confusion that I have right now is that I used a different piece of hardware to generate the video streams and the switch had no issue with dropped packets. I did the wireshark capture and IO graph just as the slides show, but even with a "good" test, it still shows the bandwidth bursting well over the line rate of the port, yet the switch show zero dropped packets. Am I missing something here?

Many thanks to anyone who takes the time to read through this and offer any advice. Mike

Joseph W. Doherty · ‎01-07-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

This multicast video stream, real-time (e.g. video conferencing) or non-real-time (e.g. IPTV)?

How many concurrent 120 Mbps inbound streams to 3560G?

How many egress multicast streams, per ingress stream.

What exact model of 3650G? What's the current running IOS version?

Little confused on recorded port drop statistics. It was but it's not now showing any port drops? "different piece of hardware"; please elaborate.

Mike Assel · ‎01-07-2013

This multicast video stream, real-time (e.g. video conferencing) or non-real-time (e.g. IPTV)?

IPTV

How many concurrent 120 Mbps inbound streams to 3560G?

There are a total of 8 streams. I'm told each stream is encoded at a variable bit rate between 12 and 20 Mbps. For this test there are two encoders generating 4 streams each and one decoder playing back all 8 streams at once.

How many egress multicast streams, per ingress stream.

For the test I was doing, there was only one egress stream per ingress.

What exact model of 3650G? What's the current running IOS version?

I don't have the exact model and IOS, but I will get it. show run says IOS 12.2.

Little confused on recorded port drop statistics. It was but it's not now showing any port drops? "different piece of hardware"; please elaborate.

Yeah, that bit is confusing. Ignore that. The important part is that on the same 3560G switch I used a different type of encoder that generated the same type an number of streams. Those streams played back fine and there was no packet loss. I took packet captures (seperately) while playing back the streams from each type of encoder. My main issue right now is interpretting the packet captures. I'm trying to make sense of the method that I got from Cisco using the Wireshark IO graph. Both the "good" and "bad" tests show the bandwidth spiking over line rate when using a 1 millisecond tick inteval (see the attached images). So I'm confused on how to prove what the issue is.

Sorry if that is convoluted. It's kind of difficult to explain. It's complex, but figured it was worth a shot on this forum. thanks

Joseph W. Doherty · ‎01-07-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Well with video, quality issues can be caused by surprisingly few drops. The fact that you were seeing some drops with the one encoder, and also had quality issues, and no drops nor quality issues for the other encoder, would imply the drops are the quality issue (as one would expect). What's curious is the difference in drops for the two different encoders for "same type" of video stream.

PS:

One interesting aspect of multicast bandwidth per stream is where packet replication consumes bandwidth.

For example, assume all gig ports, and 150 Mbps of ingress multicast. Also assume you need to "fan out" the ingress multicast to 10 ports. For the hardware architecture, where does the 1.5 Gbps (10 x 150 Mbps) of bandwidth get generated and what device bandwidth bottlenecks does it cross?