03-28-2012 08:13 AM - edited 03-07-2019 05:50 AM
We came across some weird finding when doing a show interface on one of our 3750X switch. (Please see the output below).
GigabitEthernet3/0/11 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 44d3.ca17.c40b (bia 44d3.ca17.c40b)
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 2w3d, output hang never
Last clearing of "show interface" counters 2w2d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 4294966831
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
As you can see, the interface is unused and has 0 in/out data. Yet it has countless amount of output drops. Is this a known bug or known issue with the 3750X? If I keep on doing the show interface command on that interface, I'll see it as 0. (As shown below)
GigabitEthernet3/0/11 is down, line protocol is down (notconnect)
Hardware is Gigabit Ethernet, address is 44d3.ca17.c40b (bia 44d3.ca17.c40b)
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Auto-duplex, Auto-speed, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 2w3d, output hang never
Last clearing of "show interface" counters 2w2d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts (0 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
I notice that for some reason the "Total output drops" can be unreliable as it varies depending on when I do the show command. Shouldn't it be an incremental counter that only increases and not decreases?
All our 3750X switches are using the 122-58.SE1 image. Is there any known issue with this version? Also, it seems to me that all 3750X switches tend to max out the CPU at some points, and they remain high on CPU utilization even when they're idling. Is this normal for 3750X? (See below for the history graph).
100 * * *
90 * * ** * * * *
80 * * * * ** * **** ** * * ** * *** ** *
70 **** *** ** * * ** * ** *** * ** ** **** *** ** ****** ** ** *** *** *
60 ************************************************************ *********
50 **********************************************************************
40 ************#************#********************************************
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
This particular switch is the Master SW of the stack with total of 3 members.
Please help if you have come acrossed something like this.
Thanks,
Tuan
03-28-2012 02:27 PM
All our 3750X switches are using the 122-58.SE1 image.
Dude, stay AWAY from 12.2(58)SE and 12.2(58)SE1. Go to 12.2(58)SE2 instead. SE and SE1 has an CPU hog bug and this matter has been "slightly" fixed in SE2. I say slightly because I've not seen this popping up but some people have.
03-29-2012 06:38 AM
Thank you sir!
Tuan
06-18-2012 08:04 AM
We have many 3750X stacks deployed. We have the same problem you are reporting on all stacks running 12.2(58)SE2. I consider this a bug, though I cannot locate TAC specifying it as such. Other stacks running something older than 12.2.58 are not having the problem. I have to admit, I am not used to running into bugs on edge switching -- the 3750X hasn't lived up to that standard (as we have witness/hit two other confirmed bugs - one of which is a pretty big deal).
Here is show interface output, on the same interface, taken back to back about 10 seconds apart - counters have been cleared also. (Other output parsed.) I would call that a bug....
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3157198182
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2019429068
Rick
06-18-2012 08:12 AM
edit, counters were cleared ONCE prior to running the first "show interface" command - NOT after each command afterwards... just wanted to clarify
(so the counters showing "0" output drops makes no sense, neither does the amount of drops right after a counter clearing)
08-01-2012 04:17 AM
Hi All,
We're experiencing the same issue on 12.2(58)SE2 and have reported to TAC SR#622534173
In our case this is casting doubt on the validity of the output drop counters for any interface and is likely masking the root cause of other potential issues that we're trying to get to the bottom of at the same time (if not the cause itself).
Regards,
Chris
/Edit: Refer Bug ID - CSCtq86186, further testing using the approach outlined here https://supportforums.cisco.com/docs/DOC-8093#_Toc191205670 will be used to mitigate our suspected underlying issue.
09-08-2012 06:49 AM
I have a case with TAC opened as well and we have no answers yet. We are runnign 12.2(58) SE2 and this is seriously affecting traffic flow between sites. The servers guys want to kill me.
Anyone get anywhere with this?
09-08-2012 04:26 PM
If your stack of 3750X can run 12.2(55)SE5 or SE6, use this. I've read somewhere that if the output drop counters go up and down that it could be a cosmetic bug. Output drop counters are suppose to incrementing and not drop to zero.
09-08-2012 05:40 PM
Further to my post above i've not found any reference to other bugs of relevance since...
Is the ingress port 10GbE and egress port 1GbE in your case or some other physical configuration?
Try tuning the buffers / QoS (assuming it's enabled) or otherwise augment capacity if available, etherchannel etc.
(This is where I'm at, hardware limitations permitting with some non-cisco kit on the other end however).
Failing this we may need to revert to aligning the interface speeds on ingress/egress or even changing switch platforms to something with deeper buffers perhaps.
09-09-2012 12:44 AM
Can you post the output to the following commands:
1. sh interface GIG
2. sh controller e Gig
What's the client connected to the port?
09-09-2012 09:22 AM
Here is a diagram of the network I am working on. We are having drops on the interfaces that have dotted red lines. Also here. The routers are ASR1004s. Here is the output you asked for. This from the 3750:
atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1
GigabitEthernet2/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 7081.0558.7a01 (bia 7081.0558.7a01)
Description: Connection to Edge Router ASR 01
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:54, output hang never
Last clearing of "show interface" counters 1d02h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3809940
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 1177000 bits/sec, 477 packets/sec
5 minute output rate 5340000 bits/sec, 1022 packets/sec
45498332 packets input, 14308305542 bytes, 0 no buffer
Received 4103 broadcasts (3528 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3528 multicast, 0 pause input
0 input packets with dribble condition detected
90368759 packets output, 58522200090 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
atl1c6-edgesw-3750-01#
atl1c6-edgesw-3750-01#
atl1c6-edgesw-3750-01#
atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1 controll
atl1c6-edgesw-3750-01#sh int gigabitEthernet 2/0/1 controller
GigabitEthernet2/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 7081.0558.7a01 (bia 7081.0558.7a01)
Description: Connection to Edge Router ASR 01
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:18, output hang never
Last clearing of "show interface" counters 1d02h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 3811430
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3170000 bits/sec, 657 packets/sec
5 minute output rate 4394000 bits/sec, 899 packets/sec
45734393 packets input, 14461016828 bytes, 0 no buffer
Received 4115 broadcasts (3540 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3540 multicast, 0 pause input
0 input packets with dribble condition detected
90648137 packets output, 58684921866 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Transmit GigabitEthernet2/0/1 Receive
3932381521 Bytes 2398103901 Bytes
2074042813 Unicast frames 3732423613 Unicast frames
8081455 Multicast frames 586687 Multicast frames
11782644 Broadcast frames 163346 Broadcast frames
0 Too old frames 4217137453 Unicast bytes
0 Deferred frames 73878991 Multicast bytes
0 MTU exceeded frames 10454144 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 984605609 Minimum size frames
0 8 collision frames 1179891203 65 to 127 byte frames
0 9 collision frames 591457123 128 to 255 byte frames
0 10 collision frames 235290467 256 to 511 byte frames
0 11 collision frames 98539447 512 to 1023 byte frames
0 12 collision frames 636814695 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 6575102 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
517861 64 byte frames 0 Valid frames, too small
1765696952 127 byte frames
2120331768 255 byte frames 0 Too old frames
620189651 511 byte frames 0 Valid oversize frames
88185247 1023 byte frames 0 System FCS error frames
1793765976 1518 byte frames 0 RxPortFifoFull drop frame
186753 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
atl1c6-edgesw-3750-01#
And this from the router:
atl1c6-edgert-1004-01#sh int gigabitEthernet 0/0/0 controller
GigabitEthernet0/0/0 is up, line protocol is up
Hardware is SPA-5X1GE-V2, address is 7081.05a2.9800 (bia 7081.05a2.9800)
Description: Earthlink internet facing
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 2/255, rxload 2/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is force-up, media type is LX
output flow-control is on, input flow-control is on
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:44, output 00:07:29, output hang never
Last clearing of "show interface" counters 1d20h
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 70535
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 8487000 bits/sec, 3844 packets/sec
5 minute output rate 10655000 bits/sec, 3495 packets/sec
1328015730 packets input, 456887370023 bytes, 0 no buffer
Received 76 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 2642 multicast, 0 pause input
1210406099 packets output, 394957454115 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/0/0
38 input vlan errors
0 ingress over sub drops
3 Number of sub-interface configured
atl1c6-edgert-1004-01#
I would also like to point out the average output on router interface is about 14mbits during the day. We have an MPLS circuit on there for inter-site communication.
The other router averages above 50mbits and we see no drops at all on that interface or the port-channel to the 3750.
09-09-2012 02:54 PM
SPA-5X1GE-V2
5-port 1 Gbps shared port adapter. Aside from the 3750X, how many more SFPs are using this same SPA now?
What is your ASR's ESP?
09-10-2012 05:21 AM
The only devices on the SPA-5X1GE-V2 are the provider handoff (1Gbit Internet adn MPLS circuit) and the 3750's. We have the ASR1000-ESP10 module in all our routers.
09-10-2012 03:03 PM
ESP10 should equate to around 2.5 Gbps per chassis of the ASR.
11-29-2012 04:25 AM
Same issue with 3750X and version 12.2.58(SE2), must be a cosmetic bug
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide