Output Drops between WS-C3850-48T and WS-C2960X-24PD-L

[stephen-p123]

Hi All

I've inherited a newtwork that includes the C3850 and we've installed the C2960X. These are hooked up via copper ethernet and have a very simple configuration

C3850

interface GigabitEthernet2/0/2
 description Uplink to C2960X
 switchport mode trunk
end

 C2960X

interface GigabitEthernet1/0/22
 description Uplink to C3850
 switchport mode trunk
end

On the link between C3850 and C2960 I am seeing a lot of dropped packets (Only on the C3850, nothing on the C2960) and when digging deeper, that there are a number of "Excess Defer Frames" on the C3850.

The previous team had a bunch of QOS which we tried to remove as the first step, but I suspect it exists for a reason so will look at putting it back in play.

It used to be, but was removed for diagnosis:

interface GigabitEthernet2/0/2
  description Uplink to C2960X
  switchport access vlan 77
  switchport mode trunk
  auto qos trust
  spanning-tree portfast
  service-policy input AutoQos-4.0-Trust-Cos-Input-Policy
  service-policy output AutoQos-4.0-Output-Policy
end

 The QOS settings are:

class-map match-any AutoQos-4.0-Output-Multimedia-Conf-Queue
 match dscp af41  af42  af43
 match cos  4
class-map match-any AutoQos-4.0-Output-Bulk-Data-Queue
 match dscp af11  af12  af13
 match cos  1
class-map match-any AutoQos-4.0-Output-Priority-Queue
 match dscp cs4  cs5  ef
 match cos  5
class-map match-any AutoQos-4.0-Output-Multimedia-Strm-Queue
 match dscp af31  af32  af33
class-map match-any non-client-nrt-class
class-map match-any AutoQos-4.0-Output-Trans-Data-Queue
 match dscp af21  af22  af23
 match cos  2
class-map match-any AutoQos-4.0-Output-Scavenger-Queue
 match dscp cs1
class-map match-any AutoQos-4.0-Output-Control-Mgmt-Queue
 match dscp cs2  cs3  cs6  cs7
 match cos  3
!
policy-map port_child_policy
 class non-client-nrt-class
  bandwidth remaining ratio 10
policy-map AutoQos-4.0-Output-Policy
 class AutoQos-4.0-Output-Priority-Queue
  priority level 1 percent 30
 class AutoQos-4.0-Output-Control-Mgmt-Queue
  bandwidth remaining percent 10
  queue-limit dscp cs2 percent 80
  queue-limit dscp cs3 percent 90
  queue-limit dscp cs6 percent 100
  queue-limit dscp cs7 percent 100
  queue-buffers ratio 10
 class AutoQos-4.0-Output-Multimedia-Conf-Queue
  bandwidth remaining percent 10
  queue-buffers ratio 10
 class AutoQos-4.0-Output-Trans-Data-Queue
  bandwidth remaining percent 10
  queue-buffers ratio 10
 class AutoQos-4.0-Output-Bulk-Data-Queue
  bandwidth remaining percent 4
  queue-buffers ratio 10
 class AutoQos-4.0-Output-Scavenger-Queue
  bandwidth remaining percent 1
  queue-buffers ratio 10
 class AutoQos-4.0-Output-Multimedia-Strm-Queue
  bandwidth remaining percent 10
  queue-buffers ratio 10
 class class-default
  bandwidth remaining percent 25
  queue-buffers ratio 25
policy-map AutoQos-4.0-Trust-Cos-Input-Policy
 class class-default
  set cos cos table AutoQos-4.0-Trust-Cos-Table

Port status on the C3850

C3850#sh int gi2/0/2 controller
GigabitEthernet2/0/2 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 7486.0b1a.1602 (bia 7486.0b1a.1602)
  Description: Uplink to C2960
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 5/255, rxload 4/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:11, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1225557483
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 18542000 bits/sec, 6715 packets/sec
  5 minute output rate 23283000 bits/sec, 6995 packets/sec
     42573780984 packets input, 22167505633187 bytes, 0 no buffer
     Received 791457303 broadcasts (604480626 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 604480626 multicast, 0 pause input
     0 input packets with dribble condition detected
     45485525989 packets output, 30042550904462 bytes, 0 underruns
     1225557483 output errors, 0 collisions, 1 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
Transmit                  GigabitEthernet2/0/2          Receive
30042550904462 Total bytes                      22167505633187 Total bytes
  45033860137 Unicast frames              41782323681 Unicast frames
29990443597399 Unicast bytes                    22073848041769 Unicast bytes
    320709216 Multicast frames              604480626 Multicast frames
  41076870184 Multicast bytes             79508648744 Multicast bytes
    130956636 Broadcast frames              186976677 Broadcast frames
  11030436879 Broadcast bytes             14148942674 Broadcast bytes
            0 System FCS error frames               0 IpgViolation frames
            0 MacUnderrun frames                    0 MacOverrun frames
            0 Pause frames                          0 Pause frames
            0 Cos 0 Pause frames                    0 Cos 0 Pause frames
            0 Cos 1 Pause frames                    0 Cos 1 Pause frames
            0 Cos 2 Pause frames                    0 Cos 2 Pause frames
            0 Cos 3 Pause frames                    0 Cos 3 Pause frames
            0 Cos 4 Pause frames                    0 Cos 4 Pause frames
            0 Cos 5 Pause frames                    0 Cos 5 Pause frames
            0 Cos 6 Pause frames                    0 Cos 6 Pause frames
            0 Cos 7 Pause frames                    0 Cos 7 Pause frames
            0 Oam frames                            0 OamProcessed frames
            0 Oam frames                            0 OamDropped frames
   1587370217 Minimum size frames          2414907208 Minimum size frames
  17062787834 65 to 127 byte frames       19184594389 65 to 127 byte frames
   4602431517 128 to 255 byte frames       3999123543 128 to 255 byte frames
   3063691377 256 to 511 byte frames       3114913210 256 to 511 byte frames
   2241533398 512 to 1023 byte frames      2228186107 512 to 1023 byte frames
   6983685553 1024 to 1518 byte frames     4821943045 1024 to 1518 byte frames
   9944026093 1519 to 2047 byte frames     6810113482 1519 to 2047 byte frames
            0 2048 to 4095 byte frames              0 2048 to 4095 byte frames
            0 4096 to 8191 byte frames              0 4096 to 8191 byte frames
            0 8192 to 16383 byte frames             0 8192 to 16383 byte frames
            0 16384 to 32767 byte frame             0 16384 to 32767 byte frame
            0 > 32768 byte frames                   0 > 32768 byte frames
            0 Late collision frames                 0 SymbolErr frames
  31290328555 Excess Defer frames                   0 Collision fragments
            0 Good (1 coll) frames                  0 ValidUnderSize frames
            0 Good (>1 coll) frames                 0 InvalidOverSize frames
            0 Deferred frames                       0 ValidOverSize frames
            0 Gold frames dropped                   0 FcsErr frames
            0 Gold frames truncated
            0 Gold frames successful
            0 1 collision frames
            0 2 collision frames
            0 3 collision frames
            0 4 collision frames

Reading through this, there is a rabbit hole of information and I'm honestly not sure where to start. (I do know the 3850 is old and probably should also be replaced)

Thanks

S

[balaji.bandi]

what IOS code running on both the devices.

1. have you tried replacing the physical cable and check?

2. have you tried different port the one having issue may be ?

3. when you clear the counters on the interface how soon the drops grow ?

4. Hope there is no mis match of speed on both the sides ?

5. Try simple configuration removing all the QoS on the trunk interface.

check below guide to troubleshoot :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html

[stephen-p123]

Thanks

Replaced the cable and removed the QOS information and still seeing the output drops. We have a monitoring tool that is showing the drops per min and we are also seeing issues in the real world with things dropping out.

Changed the port on the switch today and will monitor.

[balaji.bandi]

Changed the port on the switch today and will monitor.

give us the feedback how it go after moving to different port.

Replaced the cable and removed the QOS information and still seeing the output drops. We have a monitoring tool that is showing the drops per min and we are also seeing issues in the real world with things dropping out.

how soon the drops increases after clearing the counters ?

do you see any other ports also increasing the drops ?

[Joseph W. Doherty]

The Catalyst 3Ks (2Ks too) are a bit infamous for dropping bursty traffic, both due to default settings and lack of buffer RAM.  Insufficient information to say if such is your issue, but such isn't uncommon.

The extensive QoS is from auto QoS, but it can be modified.  Insufficient information to, again, say that's your case too.

If not already in your config, you might try "qos queue-softmax-multiplier 1200".

[stephen-p123]

Thanks. I'll give the cable change a 24hour window to see what it does, but if it still is causing issues will try that next. We also moved the primary device we're having issues with upstream to another switch to minimise the impact.

[stephen-p123]

Was still having issues, Tried the command and it's dropped all my connections to the switch

I think it's time to plan for a replacement..

[MHM Cisco World]

qos queue-softmax-multiplier 400 <<- apply this command only to SW face output drop not both SW

MHM

[Joseph W. Doherty]

That command wouldn't guarantee no drops, or even guarantee a reduction, much depends whether drops are due to transient or substained congestion.  It works best for the former.  The latter usually requires QoS to better manage congestion and/or more bandwidth.

BTW, was the any reduction in the rate of drops with that change?