Bursty Traffic

[salemmahara]

Hello everyone

We have Catalyst switches in our network, up-linked with 1Gb ports. Although these ports are handling less than 10-20% percent traffic at peak time, they are reporting output drops. All monitoring systems also show low traffic utilization. We thought, bursty traffic leads to this issue. 

So, here is the question:

Can we apply Qos to all ports connected to our access layer? Does it solve output drop in this situation?

How many shaping policies can be applied to a SVI or physical port (Layer 2 or 3)?

[Joseph W. Doherty]

Many of Cisco's "low-end" switches don't have huge buffer resources and their default settings don't always fully take advantage of what they do have. What can be done, that sometimes mitigates bursty traffic drops, is QoS buffer and/or queue length tuning.

These same switches often have little support for shaping, although more so for policing. You might police edge port ingress to avoid bursting an uplink into drops, but tuning the uplink's interface's configuration, is likely a better approach.

[julian.bendix]

Hi!
What kind of Catalyst Models do you have in use?

If you issue "#show interface <type> <x/y> controllers" for one of the interfaces you have drops at,
you can look at the "Excess Defer frames" Counter.
If this counter increases over time - you very likely have bursty traffic overloading your buffers.

If you have Catalyst 3650/3850 or any 9000 Series Switches,
you can enter the following command to get every Interface on the Switch to access the "shared buffer" of the Switch:

(conf)#qosqueue-softmax-multiplier 1200

This really helps a lot if you have bursty traffing going out of some Interfaces,
but it can also be bad if you have the bursty traffic going out of too many interfaces at the same time.
You could try it and look if the issues get better. (again only for Cat3650/3850/9xxx).

Best regards!

[salemmahara]

Hi Julian

Thank you for replying

This is our setup: 2960X and 3850

show int X/x controller

 MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 105/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 1w1d
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 618319748
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 921000 bits/sec, 1479 packets/sec
5 minute output rate 41306000 bits/sec, 3559 packets/sec
337423474 packets input, 95088044983 bytes, 0 no buffer
Received 36209620 broadcasts (27339872 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 27339872 multicast, 0 pause input
0 input packets with dribble condition detected
536745401 packets output, 719418675469 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet1/0/48 Receive
5060682808796 Total bytes 4850822088853 Total bytes
5422174541 Unicast frames 5763660555 Unicast frames
5037820375041 Unicast bytes 4761966513184 Unicast bytes
245121684 Multicast frames 515572822 Multicast frames
20608890830 Multicast bytes 44435093291 Multicast bytes
23108817 Broadcast frames 278542216 Broadcast frames
2253542925 Broadcast bytes 44420482378 Broadcast bytes
0 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
862959577 Minimum size frames 1563117559 Minimum size frames
1365921302 65 to 127 byte frames 1654466073 65 to 127 byte frames
109319293 128 to 255 byte frames 198950590 128 to 255 byte frames
110187956 256 to 511 byte frames 128113131 256 to 511 byte frames
88685502 512 to 1023 byte frames 34967977 512 to 1023 byte frames
3153420188 1024 to 1518 byte frames 2978191196 1024 to 1518 byte frames
0 1519 to 2047 byte frames 651 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 0 SymbolErr frames
3469488988 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
0 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

This interface is connected to a Radio link

[julian.bendix]

Hey!

Excess Defer Frames - Counter seems extremely high in your output.

Is this from the Catalyst 3850?

To my knowledge in Cat 3650 and 3850 there is a Hardware Bug which gets the Switch to show all Interface Counters in Bytes and not in Packets. If your output is from the 3850, I think we might see this Bug.

But seeing the Counter being high in the first place - means we had bursty traffic,
which overloaded the Buffers.

If we are on the 3850 - we could try adjusting the softmax buffers like I described in the last comment.

If we need to find out what the bursty traffic actually is - I don't think we can get around a SPAN Session.

Let me know how you can proceed and if you need further info.

Best regards
Julian

(P.S.: please always rate if comments are helpful and/or solved the issue)

[Joseph W. Doherty]

That link is running at 100 Mbps, not gig, and it's shows a 5 minute average of about 41 Mbps, so that's a bit busier then your OP indicated.

Your excess defer frames, as this is a full duplex connection, could further confirm bursts of traffic or overloading the transmission capacity of the interface.

[salemmahara]

Hi Joseph,

A link with 100Mb speed can handle 41Mb traffic without any problem. Am I right? 

I think this problem is caused by radio connected to the link with the capacity I just replied to Julian. 

What do you think?

[Joseph W. Doherty]

"A link with 100Mb speed can handle 41Mb traffic without any problem. Am I right?"

Maybe, maybe not. The 41 Mbps is an average usage over 5 minutes. It tells us very little about what happens at the millisecond level (which is where drops tend to occur).

[salemmahara]

Hi Julian,

This is a 3850 with Qos Soft-max configured of 1200 . 

This interface is connected to a radio which can handle 50-60Mb full duplex data and about 70-80Mb half-duplex.

Sometimes it is working at full load for about 2-3 hours to transmit data over the radio link. 

So, I think this problem is caused by radio which is not able to handle a full duplex 100Mb link. Is there any possible way to shape the traffic on 3850 and only permit 50Mb traffic over the link? 

I think the biggest problem is connecting the switch to a radio with lower capability