Outdiscards on the WS-C3750-48P 2-switch stack

N3t W0rK3r · ‎05-08-2014

Users connected to a 2-switch 3750-48P stack are reporting performance issues and voice degradation issues. This stack is connected by fibre to a Cat6509 at 1 Gig.

On the 3750 stack, all ports are showing OutDiscards incrementing, including the gig uplink to the Cat.

GigabitEthernet1/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0025.464b.649b (bia 0025.464b.649b)
Description: uplink to 6509B g1/12
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters 00:53:53
Input queue: 4/75/0/0 (size/max/drops/flushes); Total output drops: 9558
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 2744000 bits/sec, 551 packets/sec
5 minute output rate 725000 bits/sec, 408 packets/sec
1817243 packets input, 886124661 bytes, 0 no buffer
Received 306232 broadcasts (265817 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 265817 multicast, 0 pause input
0 input packets with dribble condition detected
1426219 packets output, 435026007 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

On the Cat6509, we're seeing CRC's and input errors incrementing:

GigabitEthernet1/12 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 001e.4a45.4a9b (bia 001e.4a45.4a9b)
Description: NORTH5
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is SX
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:22, output 00:00:03, output hang never
Last clearing of "show interface" counters 18:33:39
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 902000 bits/sec, 409 packets/sec
5 minute output rate 2263000 bits/sec, 573 packets/sec
16601466 packets input, 4437617260 bytes, 0 no buffer
Received 540366 broadcasts (306489 multicasts)
0 runts, 0 giants, 0 throttles
2201 input errors, 2169 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
31599818 packets output, 20927531730 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Also, on the 3750 stack, we're seeing stack link errors as follows:

May 8 08:52:37.975 EDT: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN
May 8 08:52:48.067 EDT: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP
May 8 08:52:49.082 EDT: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

We have tried changing out the SFPs, fibre patch cables, fibre runs to the Cat and even different ports on different Cats with the same results.

Not sure where else to go form here. The 3750s are running:

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE5, RELEASE SOFTWARE (fc1)

Anyone have an suggestions or further troubleshooting steps we can try to figure out exactly what's happening? The 3750s have been in operation for several years so I'm not sure if this is an age thing, or what.

Thanks in advance.

John

Naveen Venkateshaiah · ‎05-08-2014

Hi,

On the Cat6509, we're seeing CRC's and input errors incrementing: Its pure layer 1 issue.

GigabitEthernet1/12 is up, line protocol is up (connected)

2201 input errors, 2169 CRC, 0 frame, 0 overrun, 0 ignored---------->Physical Layer issue.

Action Plan:

=========

Change the cable

Change the SFP

Connect the cable bypassing the patch panel g1/12--------------other end of the Switch.

Regards,

Naveen

N3t W0rK3r · ‎05-08-2014

Thank you for your response Naveen. But as I indicated, we have already changed out SFPs, ports, fibre patches at the 3750, fibre runs to our DC, and fibre patches/ports on the CATs to no avail.

Today will use a copper SFP to uplink the 3750 stack to another 3750 stack a floor above, instead of directly to the Cat. We'll see what happens... I'll report back.

Thanks again.

Naveen Venkateshaiah · ‎05-08-2014

Hi,

You have to change fibre cable and sfp on cat 6500 side too, as g1/12 is having input and CRC errors.

Please share "sh interfaces counters errors module 1" and "sh counters interface g1/12" out put for 3 consecutive times with the interval of 60 sec each.

Regards,

Naveen

****Rate this if it is helpful****

N3t W0rK3r · ‎05-08-2014

Thank Naveen. We did change the CAT6509 end too... originally the fibre terminated on a different 6509 chassis altogether.

I have attached the output you requested.

Thanks.

John

Leo Laohoo · ‎05-08-2014

Ok, I need to get some more info from you.

1. Post the output to the 3750 command "sh controller e <BLAH>" of the port where this is having a problem and the uplink.

2. Is this issue happening happening to ALL users with voice or just this one person?

3. What IOS is both of your switch running?

I've checked the "sh counter" outputs and I'd also recommending replacing the SFP and GBIC. Prepare to replace the SFP/GBIC three (3) times. If you've replace the SFP, GBIC and fibre patch cable three times and the CRC is incrementing, move the GBIC to a different SLOT (do not use the same slot on the 6500).

NOTE: Whoever has been marking the responses with a low rating score better STOP IT. It gives the impression that the responses/recommendations are faulty. People who are searching for solution will associate a post with low rate as INCORRECT.

N3t W0rK3r · ‎05-09-2014

Thanks for all the suggestions to date.

The situation worsened last night which forced our hand. We decided to break the 3750 stack and feed each switch separately to different devices... one to a Cat6509 via fibre (with the same SFP/GBIC/patches as before) and the other to another 3750 stack one floor above, using Cat5E and copper SFPs.

Both 3750 switches are performing normally now with no degradation of service across the board.

I guess our problem was somehow related to (or caused by) the stack link errors seen in the logs.

Thanks again everyone.

John