Cisco SG500X crash - CSCuu60493 bug?

tomaszg.golebiewski · ‎08-19-2015

Hello,

according to the note:

https://tools.cisco.com/bugsearch/bug/CSCuu60493/?referring_site=bugquickviewclick

SG500x: Crash observed when port channel member link drops

I think we're dealing with this right now. This is third switch crash today. There is single interface in Port-channel, second is not available (no module inserted). 10G SFP+ single-mode link.

We have no logs, no clues what exacly caused reboot/crash since it's not registering on remote syslog server.

Can You help investigate?

tomaszg.golebiewski · ‎08-19-2015

Switch info:

show version
SW version 1.4.1.3 ( date 29-Mar-2015 time 16:25:55 )
Boot version 1.4.0.02 ( date 24-Jun-2014 time 12:01:31 )
HW version V05

Logs:

Registered on syslog:
<190>%INIT-I-Startup: Cold Startup

some other info - when switch is booting:
29-Mar-2015 18:27:07 :%LINK-I-Up: Po1
29-Mar-2015 18:27:07 :%TRUNK-I-PORTADDED: Port te1/1 added to Po1
29-Mar-2015 18:27:04 :%LINK-I-Up: te1/1
29-Mar-2015 18:27:04 :%LINK-W-Down: te1/1
29-Mar-2015 18:26:57 :%NSFP-I-SFPGibicDetected: te1/1 SFP port is present, module type - 10G BASE-SR

show fiber-ports optical-transceiver interface te1/1

Port Temp Voltage Current Output Input LOS
[C] [Volt] [mA] Power Power
[mWatt] [mWatt]
----------- ------ ------- ------- ------- ------- ---
te1/1 39 3.29 5.55 0.54 0.52 No

Temp - Internally measured transceiver temperature
Voltage - Internally measured supply voltage
Current - Measured TX bias current
Output Power - Measured TX output power in milliWatts
Input Power - Measured RX received power in milliWatts
LOS - Loss of signal
N/A - Not Available, N/S - Not Supported, W - Warning, E - Error

We're not registering CRC's on Cisco side:

show interfaces counters TengigabitEthernet 1/1

Port InUcastPkts InMcastPkts InBcastPkts InOctets
---------------- ------------ ------------ ------------ ------------
te1/1 109120522 281489 258503 146222462435

Port OutUcastPkts OutMcastPkts OutBcastPkts OutOctets
---------------- ------------ ------------ ------------ ------------
te1/1 78401104 8844 10472 42750809736

Alignment Errors: 0
FCS Errors: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

But THEY ARE on the other end:

Input Statistics:
84335121841 packets, 45191532166242 bytes
489 64-byte pkts, 55169844889 over 64-byte pkts, 256937770 over 127-byte pkts
195128419 over 255-byte pkts, 384834521 over 511-byte pkts, 28328375753 over 1023-byte pkts
9328663 Multicasts, 17951699 Broadcasts
0 runts, 0 giants, 0 throttles
20 CRC, 0 overrun, 20 discarded
Output Statistics:
118411107230 packets, 157260392978070 bytes, 0 underruns
12552657 64-byte pkts, 12195355914 over 64-byte pkts, 1678950974 over 127-byte pkts
982553697 over 255-byte pkts, 1870589561 over 511-byte pkts, 101671104427 over 1023-byte pkts
255662870 Multicasts, 198636719 Broadcasts, 117956807641 Unicasts
0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):
Input 89.00 Mbits/sec, 19613 packets/sec, 0.92% of line-rate
Output 284.00 Mbits/sec, 26989 packets/sec, 2.89% of line-rate
Time since last interface status change: 01:11:44

probably by link down condition.

tomaszg.golebiewski · ‎08-31-2015

Another crash happend 10 days later, on Sunday morning, when no one was on site.

We're about to replace FAULTY device with one running 1.3.x firmware.

Please fix this in current firmware version asap.

tomaszg.golebiewski · ‎09-09-2015

Hello all,

so today this bug has been seen on 4 x SG500X stack. Bug conditon: One of port-channels from two 1GbE ports was down.

After that it took down access to whole stack (degrading or affecting connectivity on total 96 GbE ports/devices). Some details:

- no traffic on several vlans, some were passing, some not

- port-channel down on other side (all ports, port-channel and 10GbE)

- no ICMP and dead management interface

- during Serial Cable session to SG500X master switch session died on command "show interfaces status" not even showing first interface from list, access was completly lost

- Spanning Tree on other end was ok (running RSTP)

System Stack Mode: Native Stacking
System Operational Mode: L3 and L2 Mode

Running 1.4.0.88, going to upgrade to the latest version on mainteance window, which is still affected by CSCuu60493.

tomaszg.golebiewski · ‎10-26-2015

We're now running 1.4.1.03 on affected switches.

Port-channels have all members up.

Issue still exists.

It happends even if no port-channels configured, eg. condition: 8 ports down on multi-switch SG500X stack coming from single device (storage or server). And then stack is down.

It wasn't happening on 1.3.x fw.