Cisco SG500X crash - CSCuu60493 bug?

[tomaszg.golebiewski]

Hello,

according to the note:

https://tools.cisco.com/bugsearch/bug/CSCuu60493/?referring_site=bugquickviewclick

SG500x: Crash observed when port channel member link drops

I think we're dealing with this right now. This is third switch crash today. There is single interface in Port-channel, second is not available (no module inserted). 10G SFP+ single-mode link.

We have no logs, no clues what exacly caused reboot/crash since it's not registering on remote syslog server.

Can You help investigate?

[tomaszg.golebiewski]

Switch info:

show version
SW version    1.4.1.3 ( date  29-Mar-2015 time  16:25:55 )
Boot version    1.4.0.02 ( date  24-Jun-2014 time  12:01:31 )
HW version    V05

Logs:

Registered on syslog:
<190>%INIT-I-Startup: Cold Startup

some other info - when switch is booting:
29-Mar-2015 18:27:07 :%LINK-I-Up:  Po1
29-Mar-2015 18:27:07 :%TRUNK-I-PORTADDED: Port te1/1 added to Po1
29-Mar-2015 18:27:04 :%LINK-I-Up:  te1/1
29-Mar-2015 18:27:04 :%LINK-W-Down:  te1/1
29-Mar-2015 18:26:57 :%NSFP-I-SFPGibicDetected: te1/1 SFP port is present, module type - 10G BASE-SR

show fiber-ports optical-transceiver interface te1/1

   Port      Temp  Voltage Current Output  Input   LOS
             [C]   [Volt]  [mA]    Power   Power
                                   [mWatt] [mWatt]
----------- ------ ------- ------- ------- ------- ---
   te1/1      39    3.29    5.55    0.54    0.52   No

 Temp         - Internally measured transceiver temperature
 Voltage      - Internally measured supply voltage
 Current      - Measured TX bias current
 Output Power - Measured TX output power in milliWatts
 Input Power  - Measured RX received power in milliWatts
 LOS          - Loss of signal
 N/A - Not Available, N/S - Not Supported, W - Warning, E - Error

We're not registering CRC's on Cisco side:

show interfaces counters TengigabitEthernet 1/1

      Port       InUcastPkts  InMcastPkts  InBcastPkts    InOctets
---------------- ------------ ------------ ------------ ------------
     te1/1        109120522      281489       258503    146222462435

      Port       OutUcastPkts OutMcastPkts OutBcastPkts  OutOctets
---------------- ------------ ------------ ------------ ------------
     te1/1         78401104       8844        10472     42750809736

Alignment Errors: 0
FCS Errors: 0
Multiple Collision Frames: 0
SQE Test Errors: 0
Deferred Transmissions: 0
Carrier Sense Errors: 0
Oversize Packets: 0
Internal MAC Rx Errors: 0
Symbol Errors: 0
Received Pause Frames: 0
Transmitted Pause Frames: 0

But THEY ARE on the other end:

Input Statistics:
     84335121841 packets, 45191532166242 bytes
     489 64-byte pkts, 55169844889 over 64-byte pkts, 256937770 over 127-byte pkts
     195128419 over 255-byte pkts, 384834521 over 511-byte pkts, 28328375753 over 1023-byte pkts
     9328663 Multicasts, 17951699 Broadcasts
     0 runts, 0 giants, 0 throttles
     20 CRC, 0 overrun, 20 discarded
Output Statistics:
     118411107230 packets, 157260392978070 bytes, 0 underruns
     12552657 64-byte pkts, 12195355914 over 64-byte pkts, 1678950974 over 127-byte pkts
     982553697 over 255-byte pkts, 1870589561 over 511-byte pkts, 101671104427 over 1023-byte pkts
     255662870 Multicasts, 198636719 Broadcasts, 117956807641 Unicasts
     0 throttles, 0 discarded, 0 collisions, 0 wreddrops
Rate info (interval 299 seconds):
     Input 89.00 Mbits/sec,      19613 packets/sec, 0.92% of line-rate
     Output 284.00 Mbits/sec,      26989 packets/sec, 2.89% of line-rate
Time since last interface status change: 01:11:44

probably by link down condition.

[tomaszg.golebiewski]

Another crash happend 10 days later, on Sunday morning, when no one was on site.

We're about to replace FAULTY device with one running 1.3.x firmware.

Please fix this in current firmware version asap.

[tomaszg.golebiewski]

Hello all,

so today this bug has been seen on 4 x SG500X stack. Bug conditon: One of port-channels from two 1GbE ports was down.

After that it took down access to whole stack (degrading or affecting connectivity on total 96 GbE ports/devices). Some details:

- no traffic on several vlans, some were passing, some not

- port-channel down on other side (all ports, port-channel and 10GbE)

- no ICMP and dead management interface

- during Serial Cable session to SG500X master switch session died on command "show interfaces status" not even showing first interface from list, access was completly lost

- Spanning Tree on other end was ok (running RSTP)

System Stack Mode: Native Stacking  
System Operational Mode: L3 and L2 Mode

Running 1.4.0.88, going to upgrade to the latest version on mainteance window, which is still affected by CSCuu60493.

[tomaszg.golebiewski]

We're now running 1.4.1.03 on affected switches.

Port-channels have all members up.

Issue still exists.

It happends even if no port-channels configured, eg. condition: 8 ports down on multi-switch SG500X stack coming from single device (storage or server). And then stack is down.

It wasn't happening on 1.3.x fw.