cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5811
Views
0
Helpful
15
Replies

Cisco 3750X unusual output

tuan dang
Level 1
Level 1

We came across some weird finding when doing a show interface on one of our 3750X switch.  (Please see the output below).

GigabitEthernet3/0/11 is down, line protocol is down (notconnect)

  Hardware is Gigabit Ethernet, address is 44d3.ca17.c40b (bia 44d3.ca17.c40b)

  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 2w3d, output hang never

  Last clearing of "show interface" counters 2w2d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4294966831

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     0 packets output, 0 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

As you can see, the interface is unused and has 0 in/out data.  Yet it has countless amount of output drops.  Is this a known bug or known issue with the 3750X?  If I keep on doing the show interface command on that interface, I'll see it as 0.  (As shown below)

GigabitEthernet3/0/11 is down, line protocol is down (notconnect)

  Hardware is Gigabit Ethernet, address is 44d3.ca17.c40b (bia 44d3.ca17.c40b)

  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 2w3d, output hang never

  Last clearing of "show interface" counters 2w2d

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 0 bits/sec, 0 packets/sec

     0 packets input, 0 bytes, 0 no buffer

     Received 0 broadcasts (0 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 0 multicast, 0 pause input

     0 input packets with dribble condition detected

     0 packets output, 0 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

I notice that for some reason the "Total output drops" can be unreliable as it varies depending on when I do the show command.  Shouldn't it be an incremental counter that only increases and not decreases?

All our 3750X switches are using the 122-58.SE1 image.  Is there any known issue with this version? Also, it seems to me that all 3750X switches tend to max out the CPU at some points, and they remain high on CPU utilization even when they're idling.  Is this normal for 3750X?  (See below for the history graph).

100             *            *                                           *

90             *            *          **   *        *               *  *

80          *  * *       * **       *  **** **       *  * **  * *** **  *

70 **** *** ** * * ** * ** *** * ** ** **** *** ** ****** ** ** *** *** *

60 ************************************************************ *********

50 **********************************************************************

40 ************#************#********************************************

30 ######################################################################

20 ######################################################################

10 ######################################################################

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

             0    5    0    5    0    5    0    5    0    5    0    5    0

                 CPU% per hour (last 72 hours)

                * = maximum CPU%   # = average CPU%

This particular switch is the Master SW of the stack with total of 3 members.

Please help if you have come acrossed something like this.

Thanks,

Tuan

15 Replies 15

Leo Laohoo
Hall of Fame
Hall of Fame
All our 3750X switches are using the 122-58.SE1 image.

Dude, stay AWAY from 12.2(58)SE and 12.2(58)SE1.  Go to 12.2(58)SE2 instead.  SE and SE1 has an CPU hog bug and this matter has been "slightly" fixed in SE2.  I say slightly because I've not seen this popping up but some people have.

Thank you sir!

Tuan

rsommer
Level 1
Level 1

We have many 3750X stacks deployed.  We have the same problem you are reporting on all stacks running 12.2(58)SE2.  I consider this a bug, though I cannot locate TAC specifying it as such.  Other stacks running something older than 12.2.58 are not having the problem.  I have to admit, I am not used to running into bugs on edge switching -- the 3750X hasn't lived up to that standard (as we have witness/hit two other confirmed bugs - one of which is a pretty big deal).  

Here is show interface output, on the same interface, taken back to back about 10 seconds apart - counters have been cleared also.  (Other output parsed.)  I would call that a bug....

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3157198182

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068

Rick

edit, counters were cleared ONCE prior to running the first "show interface" command - NOT after each command afterwards... just wanted to clarify

(so the counters showing "0" output drops makes no sense, neither does the amount of drops right after a counter clearing)

chris.atkinson
Level 1
Level 1

Hi All,

We're experiencing the same issue on 12.2(58)SE2 and have reported to TAC SR#622534173

In our case this is casting doubt on the validity of the output drop counters for any interface and is likely masking the root cause of other potential issues that we're trying to get to the bottom of at the same time (if not the cause itself).

Regards,

Chris

/Edit: Refer Bug ID - CSCtq86186, further testing using the approach outlined here https://supportforums.cisco.com/docs/DOC-8093#_Toc191205670 will be used to mitigate our suspected underlying issue.

I have a case with TAC opened as well and we have no answers yet. We are runnign 12.2(58) SE2 and this is seriously affecting traffic flow between sites. The servers guys want to kill me.

Anyone get anywhere with this?

If your stack of 3750X can run 12.2(55)SE5 or SE6, use this.  I've read somewhere that if the output drop counters go up and down that it could be a cosmetic bug.  Output drop counters are suppose to incrementing and not drop to zero.

Further to my post above i've not found any reference to other bugs of relevance since...

Is the ingress port 10GbE and egress port 1GbE in your case or some other physical configuration?

Try tuning the buffers / QoS (assuming it's enabled) or otherwise augment capacity if available, etherchannel etc.

(This is where I'm at, hardware limitations permitting with some non-cisco kit on the other end however).

Failing this we may need to revert to aligning the interface speeds on ingress/egress or even changing switch platforms to something with deeper buffers perhaps.

Can you post the output to the following commands:

1.  sh interface GIG ;

2.  sh controller e Gig

What's the client connected to the port?

Here is a diagram of the network I am working on. We are having drops on the interfaces that have dotted red lines. Also here. The routers are ASR1004s. Here is the output you asked for. This from the 3750:

atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1

GigabitEthernet2/0/1 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 7081.0558.7a01 (bia 7081.0558.7a01)

  Description: Connection to Edge Router ASR 01

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:54, output hang never

  Last clearing of "show interface" counters 1d02h

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3809940

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1177000 bits/sec, 477 packets/sec

  5 minute output rate 5340000 bits/sec, 1022 packets/sec

     45498332 packets input, 14308305542 bytes, 0 no buffer

     Received 4103 broadcasts (3528 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 3528 multicast, 0 pause input

     0 input packets with dribble condition detected

     90368759 packets output, 58522200090 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

atl1c6-edgesw-3750-01#

atl1c6-edgesw-3750-01#

atl1c6-edgesw-3750-01#

atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1 controll

atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1 controller

GigabitEthernet2/0/1 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 7081.0558.7a01 (bia 7081.0558.7a01)

  Description: Connection to Edge Router ASR 01

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:18, output hang never

  Last clearing of "show interface" counters 1d02h

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3811430

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 3170000 bits/sec, 657 packets/sec

  5 minute output rate 4394000 bits/sec, 899 packets/sec

     45734393 packets input, 14461016828 bytes, 0 no buffer

     Received 4115 broadcasts (3540 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 3540 multicast, 0 pause input

     0 input packets with dribble condition detected

     90648137 packets output, 58684921866 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

     Transmit GigabitEthernet2/0/1            Receive

   3932381521 Bytes                       2398103901 Bytes

   2074042813 Unicast frames              3732423613 Unicast frames

      8081455 Multicast frames                586687 Multicast frames

     11782644 Broadcast frames                163346 Broadcast frames

            0 Too old frames              4217137453 Unicast bytes

            0 Deferred frames               73878991 Multicast bytes

            0 MTU exceeded frames           10454144 Broadcast bytes

            0 1 collision frames                   0 Alignment errors

            0 2 collision frames                   0 FCS errors

            0 3 collision frames                   0 Oversize frames

            0 4 collision frames                   0 Undersize frames

            0 5 collision frames                   0 Collision fragments

            0 6 collision frames

            0 7 collision frames           984605609 Minimum size frames

            0 8 collision frames          1179891203 65 to 127 byte frames

            0 9 collision frames           591457123 128 to 255 byte frames

            0 10 collision frames          235290467 256 to 511 byte frames

            0 11 collision frames           98539447 512 to 1023 byte frames

            0 12 collision frames          636814695 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames

            0 14 collision frames                  0 Pause frames

            0 15 collision frames

            0 Excessive collisions                 0 Symbol error frames

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames            6575102 Valid frames, too large

            0 Excess defer frames                  0 Invalid frames, too small

       517861 64 byte frames                       0 Valid frames, too small

   1765696952 127 byte frames

   2120331768 255 byte frames                      0 Too old frames

    620189651 511 byte frames                      0 Valid oversize frames

     88185247 1023 byte frames                     0 System FCS error frames

   1793765976 1518 byte frames                     0 RxPortFifoFull drop frame

       186753 Too large frames

            0 Good (1 coll) frames

            0 Good (>1 coll) frames

atl1c6-edgesw-3750-01#

And this from the router:

atl1c6-edgert-1004-01#sh int gigabitEthernet 0/0/0 controller

GigabitEthernet0/0/0 is up, line protocol is up

  Hardware is SPA-5X1GE-V2, address is 7081.05a2.9800 (bia 7081.05a2.9800)

  Description: Earthlink internet facing

  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 2/255, rxload 2/255

  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set

  Keepalive not supported

  Full Duplex, 1000Mbps, link type is force-up, media type is LX

  output flow-control is on, input flow-control is on

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:44, output 00:07:29, output hang never

  Last clearing of "show interface" counters 1d20h

  Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 70535

  Queueing strategy: Class-based queueing

  Output queue: 0/40 (size/max)

  5 minute input rate 8487000 bits/sec, 3844 packets/sec

  5 minute output rate 10655000 bits/sec, 3495 packets/sec

     1328015730 packets input, 456887370023 bytes, 0 no buffer

     Received 76 broadcasts (0 IP multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 2642 multicast, 0 pause input

     1210406099 packets output, 394957454115 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

GigabitEthernet0/0/0

     38 input vlan errors

     0 ingress over sub drops

     3 Number of sub-interface configured

atl1c6-edgert-1004-01#

I would also like to point out the average output on router interface is about 14mbits during the day. We have an MPLS circuit on there for inter-site communication.

The other router averages above 50mbits and we see no drops at all on that interface or the port-channel to the 3750.

SPA-5X1GE-V2

5-port 1 Gbps shared port adapter.  Aside from the 3750X, how many more SFPs are using this same SPA now?

What is your ASR's ESP? 

The only devices on the SPA-5X1GE-V2 are the provider handoff (1Gbit Internet adn MPLS circuit) and the 3750's. We have the  ASR1000-ESP10 module in all our routers.

ESP10 should equate to around 2.5 Gbps per chassis of the ASR. 

Same issue with 3750X and version 12.2.58(SE2), must be a cosmetic bug

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: