Cisco SW 2960x Series Unexpected UpLinks Shutdown

[Herald Sison]

Hi All,,

I really need help now guys. I have 8-2960x access switches that is connected to 2-3750x core switches. The uplinks of the 2960x switches will just shutdown unexpectedly even if i just followed the same and old switch configuration.

Here is my Uplink config of 1 of my access switch and core siwtch.

Access switch:

interface Port-channel1
 switchport mode trunk
 

interface GigabitEthernet1/0/50
 description "UP-LINK01"
 switchport mode trunk
 channel-group 1 mode desirable
!
interface GigabitEthernet1/0/52
 description "UP-LINK02"
 switchport mode trunk
 channel-group 1 mode desirable

Core siwtch:

interface Port-channel2
 switchport trunk encapsulation dot1q
 switchport mode trunk

interface GigabitEthernet2/0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode desirable
!
interface GigabitEthernet2/0/2
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode desirable

Here is the stat:

#sh interfaces port-channel 1
Port-channel1 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is c472.9563.03b4 (bia c472.9563.03b4)
  MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255

Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, link type is auto, media type is unknown

I have tried setting both channels with duplex full but still shutdowns unexpectedly.

Your advise is much appreciated. I badly need help right now, my boss will be very agree if this issue will not be solved within the week. 

Thanks (bow)

[Leo Laohoo]

When the interface goes down, post the output to the following commands: 

1.  sh interface status error

2.  sh interface <BLAH>

[rakeshvelagala]

• CSCur56395

When GLC-T interfaces or SFPs are plugged into the uplink ports, the member etherchannels go down. The show idprom command gives the following error:

• CSCur94280

When GLC-T interfaces are plugged into the uplink ports, the links fail randomly.

The workaround is to use GLC-T SFP in the uplink ports and use 1G fiber SPFs. To recover from the issue, reload the switch.

[rakeshvelagala]

Upgrade the IOS to

Description:	UNIVERSAL
Release:	15.0.2a-EX5
Release Date:	19/Feb/2015
File Name:	c2960x-universalk9-mz.150-2a.EX5.bin
Min Memory:	DRAM 128 MB Flash 64 MB
Size:	17.39 MB (18233984 bytes)
MD5 Checksum:	7aa3701a30d7d5751ac158c9dde9fcad

[Herald Sison]

Hi All,

Thank you for the feedback, as of now i nominated the coreswitch as the ROOT and BRIDGE Because i found out that our router was nominated as ROOT and BRIDGE by default and it is the oldest device in the network. So far our switches uplinks are now up for 24 hours without reported error or shutdown, for now we are still observing its performance. If the problem sill still occur i will follow some steps given by you guys.

Fingers Crossedxx

Thanks you.

[rakeshvelagala]

But this do not explain why the links went down. Your config do not have any special commands but normal port channel.

We faced this issue and raised TAC. They told us of these bugs and upgrade to this version.

[Herald Sison]

Hi Sir rakeshvelagala,

The Switch uplinks went down again few minutes ago, so that does not solve the problem. and i got some errors and i captured it both ends.

Is this also caused by looping/flapping?

Thanks for your support sir.

See attached errors screen cap.

[rakeshvelagala]

Hi,

When they go down, from 2960-x switch please collect the output of below commands. Since up links are going down, I believe you will loose connectivity to the switch. Either you can console or telnet from any host if directly connected provided you have allowed it in the VTY.

#show logging

#sh interface gigabit x/x (for all the interfaces in the port channel)

#sh version

#sh int transceiver

#show idprom int gix/x (for all the interfaces in the port channel)

Once we have the outputs, we can decide whether you are hitting those bugs. Please collect all the logs before rebooting the switch.  

Thanks

[rakeshvelagala]

Hi,

Where is errors screen capture??

[Herald Sison]

Here are the screen cap of errors sir.

Thanks

[rakeshvelagala]

Hi,

Please capture the output of below commands next time the issue happens.

Thanks

[Herald Sison]

Hi Sir rakeshvelagala,

Switch uplinks is are again but only 1 switch is going down unlike last week that almost everyday 2-3 switches go down. 

I caputred an error before doing power cycle.

Please see attached.

1) authentication error even we are already inside config mode
2) SFC Fail

[rakeshvelagala · Accepted solution]

Hi,

It is confirmed you are hitting the bug!! Upgrade to below version. You have to hard reboot the switch. (Remove RPS cable also if attached to the switch, including stack cables)

Description:

UNIVERSAL

Release:

15.0.2a-EX5

Release Date:

19/Feb/2015

File Name:

c2960x-universalk9-mz.150-2a.EX5.bin

Min Memory:

DRAM 128 MB Flash 64 MB

Size:

17.39 MB (18233984 bytes)

MD5 Checksum:

7aa3701a30d7d5751ac158c9dde9fcad

[rakeshvelagala]

BUG Details

2960X: Link may go down randomly with GLC-T in uplinks

CSCur56395

Description

Symptom:
4 member etherchannel with all GLC-T between 2960X goes down on inserting a stack module with stack cable. "show idprom" gives the following error.
hulc_sfp_iic_intf_read_eeprom sfp _index 3 yeti_iic_read_retry fail

Also the stack module fails device authentication.
%ILET-1-DEVICE_AUTHENTICATION_FAIL: The FlexStack Module inserted in this switch may not have been manufactured by Cisco or with Cisco's authorization. If your use of this product is the cause of a support issue, Cisco may deny operation of the product, support under your warranty or under a Cisco technical support program such as Smartnet. Please contact Cisco's Technical Assistance Center for more information.

The issue may be seen without ether channel also.

Conditions:
When GLC-T or other SFPs are plugged into uplink ports

Workaround:
Power cycle the box

Further Problem Description:
The uplink may, at times, go link down without error. Following error will be thrown when reading idprom information after link down.

hulc_sfp_iic_intf_read_eeprom sfp _index 3 yeti_iic_read_retry fail

We may also see "%ILET-1-DEVICE_AUTHENTICATION_FAIL" error in few cases.

[Herald Sison]

Hi Sir,

Thank you for the advise. We already downloaded the said IOS version. Why did cisco released this kind of IOS with such errors on it? This is a big impact to them.

Thanks