C3850 output discards

reccon · ‎05-03-2022

We are experiencing a large number of output discards between two C3850 stacks (sw version 16.12.05b).
They are both (collapsed)core switches in two different datacenters connected by a 10G LR single-mode fiber. It’s a layer 2 connection. Almost all of these discards are in one direction A->B (not B->A)

There is a “backup” line via another switch between these sites. If we change the path costs, so the data takes the “backup path”, the output discards “shift” to the new port/path A->C (10G multi-mode SR) So, the problem has nothing to do with the fiber or the sfp module, but rather with switch A.

The data between those two switches is mainly netapp cluster data, vmotion data and surveillance cam data.

As you can see from the screenshots, the data rate is way below 10G, even below 1G.
So, we don’t have any qos configured.

I tested the connection A->B with iperf3 and I see some TCP retransmits, but no UDP loss

Interval           Transfer          Bitrate                 Retr
0.00-10.00 sec 1.09 GBytes   940 Mbits/sec   75             sender
0.00-10.04 sec 1.09 GBytes   935 Mbits/sec                  receiver

Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/906 (0%) sender
0.00-10.04 sec 1.25 MBytes 1.05 Mbits/sec 0.005 ms 0/906 (0%) receiver

Iperf3 B->A I have almost no retransmits

Interval           Transfer          Bitrate                  Retr
0.00-10.00 sec 1.09 GBytes   940 Mbits/sec    2             sender
0.00-10.04 sec 1.09 GBytes   935 Mbits/sec                  receiver

I’ve read some articles about output discards on C3850 and “microbursts”.
I tried “qos queue-softmax-multiplier 600” and “qos queue-softmax-multiplier 1200” as recommended in these articles. Unfortunately, that didn't bring any change.

Can it be microbursts?

Is there a way to tell what packets/data is being discarded?
Do we need to bother with it at all as we don't experience any problems??
Any other recommendations?

Georg Pauwen · ‎05-03-2022

Hello,

post the output of:

show interfaces x

where 'x' are the physical interfaces connecting both stacks...

reccon · ‎05-03-2022

Hello Georg,

Thanks for your reply. here you go....

Switch A:

TenGigabitEthernet2/0/17 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 6c13.d5d3.6691 (bia 6c13.d5d3.6691)
Description: stwl_core.stack3
MTU 1900 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 22928176407
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 105399000 bits/sec, 11105 packets/sec
5 minute output rate 94359000 bits/sec, 11704 packets/sec
55999612128 packets input, 72241489749272 bytes, 0 no buffer
Received 103232234 broadcasts (56430429 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 56430429 multicast, 0 pause input
0 input packets with dribble condition detected
43623313505 packets output, 39992071798227 bytes, 0 underruns
Output 165326241 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Switch B:

TenGigabitEthernet2/0/21 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 0029.c29c.ce95 (bia 0029.c29c.ce95)
Description: stwl_core.stack2
MTU 1900 bytes, BW 10000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 10Gb/s, link type is auto, media type is SFP-10GBase-LR
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/1 (size/max/drops/flushes); Total output drops: 238782713
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 86146000 bits/sec, 11012 packets/sec
5 minute output rate 105473000 bits/sec, 11040 packets/sec
79958673842 packets input, 73142820592574 bytes, 18 no buffer
Received 1191607193 broadcasts (758055528 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 758055528 multicast, 0 pause input
0 input packets with dribble condition detected
98799080857 packets output, 126621711341265 bytes, 0 underruns
Output 77306545 broadcasts (0 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎05-03-2022

@reccon wrote:

Total output drops: 22928176407

H0ly sh*t!

Leo Laohoo · ‎05-03-2022

@reccon wrote:

sw version 16.12.05b

Off topic, but can you try this command "sh platform software status con brief"?

If the memory utilization is >56% can you run further command, like "sh process memory platform sort location switch act r0"?

I just want to see the 1st page.

reccon · ‎05-03-2022

Switch A:

Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 1.34 1.29 1.34
2-RP0 Healthy 0.19 0.27 0.27
3-RP0 Healthy 0.29 0.19 0.17

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3934800 1899692 (48%) 2035108 (52%) 2762060 (70%)
2-RP0 Healthy 3934648 2012412 (51%) 1922236 (49%) 2999464 (76%)
3-RP0 Healthy 3934648 1363324 (35%) 2571324 (65%) 1892592 (48%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 14.10 3.30 0.00 82.60 0.00 0.00 0.00
1 29.85 4.70 0.00 65.33 0.00 0.10 0.00
2 30.93 4.89 0.00 64.07 0.00 0.09 0.00
3 19.30 5.40 0.00 75.20 0.00 0.10 0.00
2-RP0 0 4.00 0.60 0.00 95.40 0.00 0.00 0.00
1 2.80 0.50 0.00 96.69 0.00 0.00 0.00
2 4.29 1.19 0.00 94.50 0.00 0.00 0.00
3 5.39 1.89 0.00 92.60 0.00 0.09 0.00
4 1.39 0.19 0.00 98.40 0.00 0.00 0.00
5 1.10 0.20 0.00 98.70 0.00 0.00 0.00
3-RP0 0 2.30 0.20 0.00 97.40 0.00 0.10 0.00
1 3.20 1.20 0.00 95.50 0.00 0.10 0.00
2 0.50 0.50 0.00 99.00 0.00 0.00 0.00
3 2.29 0.29 0.00 97.40 0.00 0.00 0.00
4 0.60 0.30 0.00 99.09 0.00 0.00 0.00
5 0.69 0.19 0.00 99.10 0.00 0.00 0.00

Switch B:

Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.72 1.01 1.05
2-RP0 Healthy 0.09 0.20 0.24

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3934800 1967644 (50%) 1967156 (50%) 2781940 (71%)
2-RP0 Healthy 3934648 1960396 (50%) 1974252 (50%) 3008680 (76%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 16.81 4.30 0.00 78.87 0.00 0.00 0.00
1 18.88 4.29 0.00 76.72 0.00 0.09 0.00
2 23.92 3.00 0.00 73.07 0.00 0.00 0.00
3 22.50 4.50 0.00 73.00 0.00 0.00 0.00
2-RP0 0 5.70 1.80 0.00 92.50 0.00 0.00 0.00
1 0.70 0.30 0.00 99.00 0.00 0.00 0.00
2 3.80 0.70 0.00 95.50 0.00 0.00 0.00
3 2.00 1.10 0.00 96.89 0.00 0.00 0.00
4 1.40 0.40 0.00 98.19 0.00 0.00 0.00
5 1.80 0.00 0.00 98.19 0.00 0.00 0.00

Leo Laohoo · ‎05-03-2022

Thanks for the output.

The command "sh platform software status con brief" is a snapshot of the control-plane CPU and Memory utilization. Memory utilization of <45% is normal. Anything above 50% is not good.

Unlike classic IOS, IOS-XE memory leaks like a sieve. There is no such thing as 0% change (flat line) in the memory utilization. Make sure to monitor memory utilization on a daily basis.

reccon · ‎05-03-2022

Hi Leo

Thank you for your explanation.

We monitor memory usage via SNMP.
The utilization appears to be (almost) a flat line.
The second switch in stack A, on which the problematic port is located, has a memory usage > 50 %

Do I understand correctly that this could be a problem?

Leo Laohoo · ‎05-03-2022

@reccon wrote:

The utilization appears to be (almost) a flat line.

Modern NMS only monitor the Data Plane average of the entire stack.

The command "sh platform software status con brief" is a snapshot of the control-plane of each memory, each CPU of every switch member of the stack.

Reminder: A 3850 has four CPU (9300 has eight CPU) and a 4 Gb memory.

Let me provide a demonstration of what I am talking about. Have a look at the graph below:

NOTE: This is a stack of 3850 (3 switches), 16.12.5b, uptime of "1 year, 3 weeks"

I am seeing, for the last 250 days, an average of 30% memory utilization of the entire stack. Nothing wrong with the graph: Flat line, memory utilization at 30%. Perfectly normal.

Compare that with the one below:

This is the control-plane memory utilization of a switch member of the same switch stack above. This tells me a completely different story.

Remember, 80% of IOS-XE memory leak and CPU hogs always occur in the control-plane.

NOTE:

Orange "box" is a normal memory utilization.
RED "box" is due to the QuoVadis Certificate issue (FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality)
Green "box" is a few days after I applied the FN-72323 workaround. Notice the memory leak has tapered?

In your response, you provided two screenshots. The small screenshot is what I am talking about. Monitor that because that is the control plane. Make sure the memory utilization does not go above 55%. Anything above that is, potentially, trouble.

This response does not address the issue of the high amount of Packet Output Drops but I want to share my experience with monitoring the control-plane of every platform running IOS-XE. It does not matter if the platform is a router, switch, WLC or a peanut-butter sandwich, as long as it runs on IOS-XE, monitor the control-plane. Daily.

Hope this helps.

Georg Pauwen · ‎05-03-2022

Hello,

the 3850 should support the command 'qos queue-softmax-multiplier 1200', that command often helps with output drops...

reccon · ‎05-03-2022

I tried “qos queue-softmax-multiplier 600” and “qos queue-softmax-multiplier 1200” as recommended.

Unfortunately, that didn't bring any change.

MHM Cisco World · ‎05-03-2022

show interface
MTU 1900 bytes <- why MTU is 1900 are you misconfig the MTU ?

reccon · ‎05-03-2022

The switch was preconfigured with MTU 1900 by our supplier.
We didn't change that.

The access switches have a MTU of 1500.
The core switch with a higher MTU than the access switches shouldn't be a problem, right?

And in this case both ends (switch A and B) have the same MTU. If that would be the problem, we should see it in both directions, shouldn't we?

MHM Cisco World · ‎05-03-2022

SW# show system mtu

use ping in PC connect to one end and use MTU 1850 do you success or not??

reccon · ‎05-05-2022

Sorry for the late reply. Didn't have time to test it yesterday.

Ping to the other side with mtu=1850 and do not fragment bit set, on a Ubuntu client directly connected to the 3850 switch, succeeded.