High Number of Out Discards on 2960x Switches

[Dave Cancela]

Hi. I first would like to point out that i did not configure these switches, i inherited them. So here goes.

We are getting a lot of out discard errors on all our edge switches. Our edge switches are all 2960x's and are stacked (there are about 3 or 4 per stack). We have 11 edge switch locations. I believe it has to do with our qos setup.

We currently have mls qos enabled and here is the config for that portion:

!
mls qos map cos-dscp 0 8 16 24 32 46 46 56
mls qos srr-queue output cos-map queue 1 threshold 3 4 5
mls qos srr-queue output cos-map queue 2 threshold 1 2
mls qos srr-queue output cos-map queue 2 threshold 2 3
mls qos srr-queue output cos-map queue 2 threshold 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output cos-map queue 4 threshold 3 1
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33 40 41 42 43 44 45
mls qos srr-queue output dscp-map queue 1 threshold 3 46 47
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31 34 35
mls qos srr-queue output dscp-map queue 2 threshold 1 36 37 38 39
mls qos srr-queue output dscp-map queue 2 threshold 2 24
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
mls qos queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 2 125 125 100 400
mls qos queue-set output 1 threshold 3 100 100 100 400
mls qos queue-set output 1 threshold 4 60 150 50 200
mls qos queue-set output 1 buffers 15 25 40 20
mls qos
!

Here is one of the ports that have the issue:

Port           Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi2/0/30    0                 0               0                0            0                 13005659

Port Configuration:

Current configuration : 284 bytes
!
interface GigabitEthernet2/0/30
 switchport trunk native vlan 120
 switchport mode trunk
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 auto qos trust
 storm-control broadcast level 60.00
 spanning-tree portfast
 spanning-tree bpduguard enable
end                          

GigabitEthernet2/0/30 is down, line protocol is down (notconnect)
  Hardware is Gigabit Ethernet, address is 38ed.1824.e61e (bia 38ed.1824.e61e)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 1w0d, output hang never
  Last clearing of "show interface" counters 4w2d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 13005659
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     12505 packets input, 2199739 bytes, 0 no buffer
     Received 1709 broadcasts (1008 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1008 multicast, 0 pause input
     0 input packets with dribble condition detected
     12616125 packets output, 2153347204 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Any ideas?

And yes, the workstation is currently unplugged from the port at the moment. Duplex and Speed are set to auto, although i have noticed the wrong port speed gets assigned sometimes. In this case, it is set to full duplex and 1000 speed.

[Robert Hillcoat]

Can you run this command on the interface please?

show mls qos interface gigabitethernet2/0/30 queueing

The strict priority-queue out will be serviced first so you need to be sure its just voice traffic thats in that queue. That could be a possible reason why your seeing drops. 

What exactly is usually plugged into that port? Just a PC or a PC and VoIP phone? 

[Dave Cancela]

Heres the result to the command:

GigabitEthernet2/0/30
Egress Priority Queue : enabled
Shaped queue weights (absolute) :  25 0 0 0
Shared queue weights  :  1 30 35 5
The port bandwidth limit : 100  (Operational Bandwidth:100.0)
The port is mapped to qset : 1

This specific port is just a Mac workstation. We have some ports that have pass-through on the phone for their machine. But this port is just a PC.

[InayathUlla Sharieff]

Hi Dave,

were there any reboot of any of the switch in the stack?

CSCur20551
Increase OutDiscards counter when one side of stack switch is reloaded.

[Dave Cancela]

No, no reboots. Uptime is about 2 months on them.

[Dave Cancela]

In case anyone else runs into this problem, I figured it out. It was caused by spanning tree constantly (every 20 seconds) flushing the mac tables and rebuilding them.

If  you run the command sh spann det | inc ieee |occur|from|isexec you can see how long since the last changes were made. If your time frame is minutes, or hours, then you might be running into the same problem as me. My core stack was at 99% cpu all the time. That is what led me to this problem. 

Check your spanning tree settings on your endpoint interfaces. Please made sure all your endpoint interfaces are setup correctly with portfast (for access) or portfast trunk (for trunk). Also make sure you do not have portfast on interfaces connected to other switches.

Hope this helps anyone else with this problem. All my packet loss and latency issues are gone.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Thank you for posting the follow-up.

In your original post you have:

interface GigabitEthernet2/0/30
 switchport trunk native vlan 120
 switchport mode trunk
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust dscp
 auto qos trust
 storm-control broadcast level 60.00
 spanning-tree portfast
 spanning-tree bpduguard enable
end                          

As this port is a trunk, normal portfast doesn't apply, a subtle issue, which you mention in your follow-up.  Even so, often most user edge ports aren't trunk ports, so this probably wouldn't be an obvious line of pursuit.

I only mention this to highlight how difficult it can be to help resolve issues with partial information and/or why those trying to help sometimes ask questions that appear off-topic.

[Robert Hillcoat]

Well if you don't need to use QOS then just disable it. What essentially doing here is trusting any markings the MAC book sends you. It might have more of an inadvertent effect further along the path i.e. the WAN interface. 

[Dave Cancela]

We do need QoS unfortunately.

So i turned off mls qos from a 2960x switch stack i have for testing, and i was still getting the high out discards. Any reason why this may be?

[Robert Hillcoat]

I meant disable it on a per port basis. 

[Dave Cancela]

Yes, i know. It didn't make a difference whether port based or globally is what i meant to imply. Not sure why though.

[Robert Hillcoat]

You have the command "priority-queue out" remove this because you don't need a priority queue on a port with only a PC connected. 

[Dave Cancela]

I did notice that as well and removed it. I am still getting about 12% packet loss on the port though.

[Robert Hillcoat]

have you cleared your counters? 

[Dave Cancela]

I did, and i still see drops. However I now also notice that is auto configured to 10Mb Half-Duplex !! What the heck is going on...