C3850 output drops

[Mika J]

Version: 3.06.06E

Platform: WS-C3850-48U

Hello,

I'm seeing output drops on an interface connected to a Wifi AP

The switch uplink is 10G and the AP is connected to 2x1gi => Po

N9K <-- Po=2x10G --> C3850 <-- Po=2x1G --> AP2802i

My understanding is there's a kind of speed mismatch and too many packets want to exit the port.

I tried to reallocate buffers but I still see these drops (see end of the post for configuration).

The traffic has no DSCP and the second port in the Po has no error

Users seem to feel disconnections even when the traffic is not high 10Mbit/s (or even lower)

What am I missing, I read the Cisco documentations

On the switch

#show int gi6/0/24 | i drops|error
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1569626
     1569626 output errors, 0 collisions, 0 interface resets

No other error such as CRC, collisions

#show int gi6/0/24  counters errors
Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi6/0/24            0           0     1569626           0          0      1569626

#show policy-map int gi6/0/24
 GigabitEthernet6/0/24

  Service-policy output: NODROP
    Class-map: class-default (match-any)
      0 packets
      Match: any
      Queueing
      (total drops) 1569626
      (bytes output) 33589434502
      bandwidth 100% (1000000 kbps)
      queue-buffers ratio 100

#show platform qos queue config gi 6/0/24
DATA Port:6 GPN:956 AFD:Disabled QoSMap:1 HW Queues: 48 - 55
  DrainFast:Disabled PortSoftStart:2 - 10000
----------------------------------------------------------
  DTS Hardmax   Softmax  PortSMin GlblSMin  PortStEnd
  --- --------  -------- -------- --------- ---------
 0   1  4     0  8 10000  7   800   3   300   2 10000

 Priority   Shaped/shared   weight  shaping_step
 --------   ------------   ------  ------------
 0      7     Shared            50           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- -------  ------  ------  ------  ------  ------  ------ ------
 0      0    7968       0       0    8906       0       0   10000       0

#show platform qos queue stats gi 6/0/24
DATA Port:6 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
    0       0           0           0 33589711669

DATA Port:6 Drop Counters
-------------------------------
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop
----- ----------- ----------- ----------- ----------- -----------
    0           0           0     1569626           0           0

interface GigabitEthernet6/0/24
 switchport access vlan 10
 switchport mode access
 storm-control broadcast level 1.00 0.50
 storm-control action trap
 channel-group 4 mode active
 spanning-tree portfast
 service-policy output NODROP
end

policy-map NODROP
 class class-default
  bandwidth percent 100
  queue-buffers ratio 100

[Leo Laohoo]

It's a known bug, CSCvb65304.

[David Nolasco]

Hi  Mika,

Please post the following output:

show run | i  softmax

if softmax multipler is set to 100, increase it up to 1200 and let us know how it goes.

Regards

David

[Mika J]

Hello,

Thank you for your answers

David: I forgot to mention that I already applied

qos queue-softmax-multiplier 1200

Leo: I have read this bug but it looks like a comsmetic bug whereas the drops seems to be real. However I plan to upgrade to 3.06.07b

The WLC and AP are in version 8.5.110.0

#sh interface Gi6/0/24 contr
GigabitEthernet6/0/24 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 50f7.2778.1888 (bia 50f7.2778.1888)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:15, output never, output hang never
  Last clearing of "show interface" counters 7w1d
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1569626
  Queueing strategy: Class-based queueing
  Output queue: 0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 1000 bits/sec, 1 packets/sec
     43816153 packets input, 17313295590 bytes, 0 no buffer
     Received 364800 broadcasts (364800 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 364800 multicast, 0 pause input
     0 input packets with dribble condition detected
     50776790 packets output, 33407289779 bytes, 0 underruns
     1569626 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet6/0/24                 Receive
  51170201466 Total bytes                 17510621725 Total bytes
     69965681 Unicast frames                 44418999 Unicast frames
  50400320917 Unicast bytes               17417333386 Unicast bytes
      3305276 Multicast frames                 459664 Multicast frames
    332310634 Multicast bytes                93286269 Multicast bytes
      1974695 Broadcast frames                     10 Broadcast frames
    437569915 Broadcast bytes                    2070 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
      4080117 Minimum size frames                 115 Minimum size frames
      4526128 65 to 127 byte frames           6739021 65 to 127 byte frames
     32971017 128 to 255 byte frames         27367843 128 to 255 byte frames
      3139495 256 to 511 byte frames          1944821 256 to 511 byte frames
      1270966 512 to 1023 byte frames         1394208 512 to 1023 byte frames
     29257929 1024 to 1518 byte frames        7432665 1024 to 1518 byte frames
            0 1519 to 2047 byte frames              0 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
      1569626 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames
            0 5 collision frames
            0 6 collision frames
            0 7 collision frames
            0 8 collision frames
            0 9 collision frames
            0 10 collision frames
            0 11 collision frames
            0 12 collision frames
            0 13 collision frames
            0 14 collision frames
            0 15 collision frames
            0 Excess collision frames
LAST UPDATE 710 msecs AGO

[Leo Laohoo]

---

@Mika J wrote:

Leo: I have read this bug but it looks like a comsmetic bug whereas the drops seems to be real. However I plan to upgrade to 3.06.07b 

---

Total Output Drops is exactly equal to Transmit output errors, which, is actually due to Excessive Deferred Frames.  

And I've seen this.  I suspect this is due to QoS which is enabled.  

[Mika J]

Hello,

On the C3850 platform QoS cannot be disabled.

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/118629-technote-qos-00.html#anc14

I don't get why a C3850 would drop traffic with such a small load

[David Nolasco]

Please post the following outputs from both interfaces that belong to port-channel:

show platform qos queue stat gig 6/0/24

show platform qos queue config gig 6/0/24

[Mika J]

Hello David,

Here's the complete output

#show platform qos queue stat gig 6/0/24
DATA Port:6 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
    0       0           0           0 33984241356
    1       0           0           0           0
    2       0           0           0           0
    3       0           0           0           0
    4       0           0           0           0
    5       0           0           0           0
    6       0           0           0           0
    7       0           0           0           0
DATA Port:6 Drop Counters
-------------------------------
Queue Drop-TH0    Drop-TH1    Drop-TH2    SBufDrop    QebDrop
----- ----------- ----------- ----------- ----------- -----------
    0           0           0     1569626           0           0
    1           0           0           0           0           0
    2           0           0           0           0           0
    3           0           0           0           0           0
    4           0           0           0           0           0
    5           0           0           0           0           0
    6           0           0           0           0           0
    7           0           0           0           0           0
 AQM Broadcast Early WTD COUNTERS(In terms of Bytes)
--------------------------------------------------
  PORT TYPE          ENQUEUE             DROP
--------------------------------------------------
 UPLINK PORT-0        N/A               0
 UPLINK PORT-1        N/A               0
 UPLINK PORT-2        N/A               0
 UPLINK PORT-3        N/A               0
 NETWORK PORTS           0                  0
 RCP PORTS               0                  0
 CPU PORT                0                  0
Note: Queuing stats are in bytes

#show platform qos queue config gig 6/0/24
DATA Port:6 GPN:956 AFD:Disabled QoSMap:1 HW Queues: 48 - 55
  DrainFast:Disabled PortSoftStart:2 - 10000
----------------------------------------------------------
  DTS Hardmax   Softmax  PortSMin GlblSMin  PortStEnd
  --- --------  -------- -------- --------- ---------
 0   1  4     0  8 10000  7   800   3   300   2 10000
 1   1  4     0  5     0  5     0   0     0   2 10000
 2   1  4     0  5     0  5     0   0     0   2 10000
 3   1  4     0  5     0  5     0   0     0   2 10000
 4   1  4     0  5     0  5     0   0     0   2 10000
 5   1  4     0  5     0  5     0   0     0   2 10000
 6   1  4     0  5     0  5     0   0     0   2 10000
 7   1  4     0  5     0  5     0   0     0   2 10000
 Priority   Shaped/shared   weight  shaping_step
 --------   ------------   ------  ------------
 0      7     Shared            50           0
 1      0     Shared         10000         128
 2      0     Shared         10000         118
 3      0     Shared         10000           0
 4      0     Shared         10000         144
 5      0     Shared         10000           0
 6      0     Shared         10000         128
 7      0     Shared         10000           0

   Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
   ------- -------  ------  ------  ------  ------  ------  ------ ------
 0      0    7968       0       0    8906       0       0   10000       0
 1      0       0       0       0       0       0       0       0       0
 2      0       0       0       0       0       0       0       0       0
 3      0       0       0       0       0       0       0       0       0
 4      0       0       0       0       0       0       0       0       0
 5      0       0       0       0       0       0       0       0       0
 6      0       0       0       0       0       0       0       0       0
 7      0       0       0       0       0       0       0       0       0

[Georg Pauwen]

Hello,

on a side note, you could try to increase the hold queue for the interface:

interface GigabitEthernet6/0/24

hold-queue 250 out

Also, I would get rid of the service policy on the interface altogether, everything goes to the default class anyway, so it doesn't do much except using up CPU cycles...

[Mika J]

Hello,

I've set a few ports with that configuration and will observe the results.

Thank you

[hborresen]

Did it work?  I have a similar problem

[Georg Pauwen]

Hello,

can you elaborate ? Do you experience output drops ?

[casanavep]

Similar Problem, but no QoS applied:

SwitchA#sh ver
Cisco IOS XE Software, Version 16.06.03
Cisco IOS Software [Everest], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.6.3, RELEASE SOFTWARE (fc8)

SwitchA#sh int te1/0/10

TenGigabitEthernet1/0/10 is up, line protocol is up (connected)

  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 22822449

  44871012 packets output, 46668260489 bytes, 0 underruns
  0 output errors, 0 collisions, 0 interface resets

SwitchA#sh cdp ne te1/0/10 de

Device ID: SwitchB
Entry address(es):
IP address: 10.125.32.100

SwitchA#ping
Protocol [ip]:
Target IP address: 10.125.32.100
Repeat count [5]: 500
Datagram size [100]: 1500
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 500, 1500-byte ICMP Echos to 10.125.32.100, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!
Success rate is 100 percent (500/500), round-trip min/avg/max = 1/3/24 ms
SwitchA#

show platform hardware fed switch active qos queue stats interface tenGigabitEthernet 1/0/10

DATA Port:1 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0     0        0        0        0        0
1     0        0        44058590 0        0

[Georg Pauwen]

Hello,

you have two options: either configure a QoS policy and increase the queue buffers, or, use the command below (in global configuration mode):

qos queue-softmax-multiplier 1200

I would suggest to use that command first, the drops should go to zero...

[casanavep]

I have implemented the global command, but output drops are still increasing.  I am not sure if this is bug: CSCvd38417; so, the buffer change won't take place until reboot.  Thoughts?  I can schedule a reload at 4:00am.