3750 High output drops and performance issues

n · ‎07-20-2015

I am experiencing performance issues with 3750 12.2(50)SE3.

On some interfaces transmit utilization is 100% and have a lot of output drops. It doesn't depend on if it is IPTV, VoIP or normal Access point. As a temporary solution i reconnect cable to unused port and copy same config to a new port.

LSW-11_02#sh ver
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(50)SE3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 22-Jul-09 06:19 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02900000

ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

PAAC-HO-LSW-11_02 uptime is 2 weeks, 6 days, 19 hours, 36 minutes
System returned to ROM by power-on
System restarted at 18:59:03 ACHGABAT Mon Jun 29 2015
System image file is "flash:c3750-ipbasek9-mz.122-50.SE3/c3750-ipbasek9-mz.122-50.SE3.bin"

n · ‎07-20-2015

LSW-11_02#sh interfaces f3/0/9
FastEthernet3/0/9 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0026.523f.4e0b (bia 0026.523f.4e0b)
Description: * IPTV *
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 247/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 758813644
Queueing strategy: fifo
Output queue: 0/0 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 98768000 bits/sec, 9071 packets/sec
     272295 packets input, 25607925 bytes, 0 no buffer
     Received 173115 broadcasts (173070 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 173070 multicast, 0 pause input
     0 input packets with dribble condition detected
     5783600419 packets output, 7849261836953 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

ej8692300 · ‎07-20-2015

See if this has anything relevance here:

https://tools.cisco.com/quickview/bug/CSCtq86186

n · ‎07-20-2015

CSCtq86186 bug is only shows incorrectly output drops but i have also issues with performance.

Madhukrishnan Gopinathan Nair · ‎07-20-2015

Hello,

Looking at the above output, it is clear that the interface is taking more than what it physically send over the wire. If the utilization that high, you cannot do anything than changing to a gig port.

However one thing i clearly notice is the below

5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 98768000 bits/sec, 9071 packets/sec

This makes me believe that there could be some kind of flooding happening on the network. Check for any interface whose input rate is abnormally high and try to shut it down if it is not expected to send that much traffic. Also check for spanning tree stability. Enable mac move notification to check if any loop is occurring.

conf t# mac address table notification mac-move

Hope this helps you,

Madhu.

****rating encourages participation*****

Madhukrishnan Gopinathan Nair · ‎07-20-2015

One more thing is you can try to sniff using SPAN to see what it is sending out. Probably it may give us information about what is the source of it.

Madhu

****rating encourages participation***

n · ‎07-20-2015

LSW-11_02#show run interface f3/0/9
interface FastEthernet3/0/9
description * IPTV *
switchport trunk encapsulation dot1q
switchport trunk native vlan 300
switchport trunk allowed vlan 10,300
switchport mode trunk
switchport nonegotiate
ip arp inspection limit rate 30
srr-queue bandwidth share 1 70 25 5
srr-queue bandwidth shape 3 0 0 0
priority-queue out
storm-control broadcast level 1.00
storm-control action shutdown
no cdp enable
spanning-tree portfast trunk
service-policy input QOS_POLICY

n · ‎07-20-2015

LSW-11_02#sh controllers ethernet-controller f3/0/9

     Transmit FastEthernet3/0/9               Receive
   3577533299 Bytes                         25640553 Bytes
       351373 Unicast frames                   99342 Unicast frames
   1499002000 Multicast frames                173179 Multicast frames
     12275487 Broadcast frames                    45 Broadcast frames
            0 Too old frames                14226181 Unicast bytes
            0 Deferred frames               11411492 Multicast bytes
            0 MTU exceeded frames               2880 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames               36207 Minimum size frames
            0 8 collision frames              229752 65 to 127 byte frames
            0 9 collision frames                   0 128 to 255 byte frames
            0 10 collision frames                  0 256 to 511 byte frames
            0 11 collision frames                  2 512 to 1023 byte frames
            0 12 collision frames               6605 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
      2636179 64 byte frames                       0 Valid frames, too small
     16720917 127 byte frames
      2234964 255 byte frames                      0 Too old frames
       329764 511 byte frames                      0 Valid oversize frames
       315863 1023 byte frames                     0 System FCS error frames
   1489389486 1518 byte frames                     0 RxPortFifoFull drop frame
         1687 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

n · ‎07-20-2015

LSW-11_02#sh mls qos
QoS is disabled
QoS ip packet dscp rewrite is enabled

n · ‎07-20-2015

Uplink is :

LSW-11_02#sh interfaces gi1/0/1 | i tx
reliability 255/255, txload 1/255, rxload 66/255

Any kind of help is appreciated.

Joseph W. Doherty · ‎07-20-2015

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

The 3560/3750 (and 2960 too) series is a bit infamous for dropping packets on congested ports. This because these switches don't have a lot of buffer RAM resources, and how they manage it, can be very suboptimal in some situations.

The 3750 has 2 MB of RAM per 24 copper ports and per its uplink ports, so placement of "busy" ports can be important.

Enabling QoS, but running with its default settings, often leads to many drops. You note in another posting that QoS is disabled, which is often a good first step, if QoS isn't needed, but I've found you can sometimes get the best results if you enable QoS and manually tune the buffer settings.

If only some ports are busy, and bursty, I've found setting up buffering to work from the common pool can often greatly reduce drops.

n · ‎07-27-2015

We found out that problem was only with IPTV(multicast packets).

For unknown reason core swtich(6509) doesn't query with igmp snooping querier and swtiches just flood all joined multicast groups' packets to interfaces.

We have enabled ip igmp snooping querier on all access layer switches solve this problem.

Thanks all for your suggestions.