Cisco WS-C2960X-48FPD-L switches goes to down automatically

[sangeethsan]

Hi ,

There was a strange device outage happed in my environment. Suddenly three switches goes to down , but there was not power failure in the data center. Based on my trouble shooting I found that the switches were WS-C2960X-48FPD-L. 3 nos. reset of the devices were running without fail.

I have done some trouble shooting steps

- Power code unplugged and plugged nearly 3-5 times in periodically and switches not came upSwitching, Catalyst 2000, Other SwitchesSwitching, Catalyst 2000, Other Switches

- power cable unplugged for more than 2 hour

- After 2 hour power cable  plugged and switches came up immediately.

The behavior of the switch was so surprised to me. This kind of issue I have  faced first time in my life.

Anybody can answer, what is the route cause of this outage. Is it any bug this particular series switch ?

Same time other switches ,router everything working as usual.

[marce1000]

                                   >what is the route cause of this outage

  Almost certainly 'external' cisco devices never behave that way. To follow-up on their experience in your network configure syslogging to a central syslog server and send snmp traps to some snmp manager software. Keep an eye on devices reporting strange events related to environmental conditions (e.g.)

 M.

[sangeethsan]

Hi marce1000,

In NMS i can see only device down/up snmp traps only there.

[Leo Laohoo]

I need to see the complete output to the command "sh log on uptime detail".

[sangeethsan]

Hi,

ACS-SW-30115#show log on uptime detail
--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 11/10/2015 06:22:12
Total uptime : 0 years 17 weeks 0 days 16 hours 35 minutes
Total downtime : 1 years 39 weeks 1 days 23 hours 51 minutes
Number of resets : 10
Number of slot changes : 0
Current reset reason : 0x0
Current reset timestamp : 05/25/2021 20:30:57
Current slot : 1
Current uptime : 0 years 0 weeks 0 days 0 hours 0 minutes
--------------------------------------------------------------------------------
Reset | |
Reason | Count |
--------------------------------------------------------------------------------
No historical data to display
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
11/10/2015 06:22:12 0x0 0 0 0 0 0
04/21/2016 06:40:17 0x0 0 0 0 1 0
04/21/2016 09:35:35 0x0 0 0 0 2 0
07/24/2017 11:36:32 0x0 0 0 0 3 0
07/28/2017 04:48:56 0x0 0 0 0 2 0
07/28/2017 13:44:17 0x0 0 0 0 8 0
08/11/2017 11:02:43 0x0 0 0 0 0 35
09/22/2017 11:34:29 0x0 0 5 6 19 0
12/04/2017 05:01:25 0x0 0 10 2 16 0
12/08/2017 22:49:00 0x0 0 0 4 13 0
--------------------------------------------------------------------------------

[Leo Laohoo]

1. I do not see any signs of software errors or crash.  
2. OBFL is of no help possibly due to the use of an "interim" firmware. 
3. Switch is running an "EX" firmware.  Strongly recommend upgrading to something more "recent".  Any version would be better than "EX".

[marce1000]

             >In NMS i can see only device down/up snmp traps only there.

 - Check your powering facilities and or provide dedicated power at all times.

 M.

[sangeethsan]

Hi Marce,

Dedicated power is there. there was  no power outage on that time. rest all were working fine during this incident 

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
dir flash1:
dir flash2:
dir flash3:
sh log on up details
remote command 2 sh log on up details
remote command 3 sh log on up details

[sangeethsan]

Hi Leo,

Few commands are not accepting . rest of the outputs are here 

ACS-SW-30115#sh version
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2a)EX5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 16-Feb-15 08:16 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(2r)E2, RELEASE SOFTWARE (fc1)

ACS-SW-30115 uptime is 1 day, 10 hours, 46 minutes
System returned to ROM by power-on
System restarted at 20:24:22 UTC Tue May 25 2021
System image file is "flash:/c2960x-universalk9-mz.150-2a.EX5/c2960x-universalk9-mz.150-2a.EX5.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-48FPD-L (APM86XXX) processor (revision K0) with 524288K bytes of memory.
Processor board ID xxxxxxxxxxxxxxxxx
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
50 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : xxxxxxxxxxxxxxxxx
Motherboard assembly number : 73-15964-03
Power supply part number : 341-0527-01
Motherboard serial number : xxxxxxxxxxxxxxxxx
Power supply serial number : xxxxxxxxxxxxxxxxx
Model revision number : K0
Motherboard revision number : A0
Model number : WS-C2960X-48FPD-L
Daughterboard assembly number : 73-14200-03
Daughterboard serial number : xxxxxxxxxxxxxxxxx
System serial number : xxxxxxxxxxxxxxxxx
Top Assembly Part Number : 800-41461-02
Top Assembly Revision Number : A0
Version ID : V03
CLEI Code Number : CMMLJ00ARC
Daughterboard revision number : A0
Hardware Board Revision Number : 0x12

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 52 WS-C2960X-48FPD-L 15.0(2a)EX5 C2960X-UNIVERSALK9-M

Configuration register is 0xF

ACS-SW-30115#dir flash1:
Directory of flash:/

2 -rwx 1456 Dec 2 2020 09:28:41 +00:00 vlan.dat
3 -rwx 2072 May 25 2021 20:54:07 +00:00 multiple-fs
4 -rwx 6252 May 25 2021 20:54:07 +00:00 config.text
5 -rwx 1916 May 25 2021 20:54:07 +00:00 private-config.text
6 drwx 512 Sep 6 2015 00:07:54 +00:00 c2960x-universalk9-mz.150-2a.EX5
645 drwx 512 Sep 6 2015 00:07:55 +00:00 dc_profile_dir

122185728 bytes total (98696704 bytes free)
ACS-SW-30115#
SBSTJ_ACS-SW-30115#dir flash2:
^
% Invalid input detected at '^' marker.

ACS-SW-30115#dir flash2: ?
% Unrecognized command
ACS-SW-30115#dir flash3:
^
% Invalid input detected at '^' marker.

ACS-SW-30115#dir flash3:
^
% Invalid input detected at '^' marker.

ACS-SW-30115#sh log on up details ?
% Unrecognized command
ACS-SW-30115#sh log ?
count Show counts of each logging message
history Show the contents of syslog history table
onboard Onboard logging information
persistent Show the contents of the logging persistent
xml Show the contents of XML logging buffer
| Output modifiers
<cr>

ACS-SW-30115#remote comm
ACS-SW-30115#remote command ?
<1-8> Switch number
all All switches

ACS-SW-30115#remote command 2 sh log on up details ?
LINE <cr>

ACS-SW-30115#remote command 2 sh log on up details
Remote switch 2 is not ready/present

ACS-SW-30115#remote command 3 sh log on up details
Remote switch 3 is not ready/present

ACS-SW-30115#
ACS-SW-30115#sh log on up details
^
% Invalid input detected at '^' marker.