4503-E Keeps rebooting for Unknown Reason

Amro BaniHani · ‎08-21-2013

Dear All,

Could you please help in finiding why my switch keeps rebooting every 8-12 hours , so please please find below some helpful output waiting for your kind help

Show version

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 12.2(54)SG1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 27-Jan-11 12:19 by prod_rel_team

Image text-base: 0x10000000, data-base: 0x12C41C68

ROM: 12.2(44r)SG5

Darkside Revision 4, Nexu Revision 9, Fortooine Revision 1.22

SCHQ-DSW-02 uptime is 6 hours, 22 minutes

System returned to ROM by reload

System restarted at 10:54:47 UAE Wed Aug 21 2013

System image file is "bootflash:cat4500e-ipbasek9-mz.122-54.SG1.bin"

Last reload reason: Unknown reason

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C4503-E (MPC8548) processor (revision 8) with 524288K bytes of memory.

Processor board ID SPE135100LX

MPC8548 CPU at 1GHz, Supervisor 6L-E

Last reset from Reload

2 Virtual Ethernet interfaces

64 Gigabit Ethernet interfaces

8 Ten Gigabit Ethernet interfaces

511K bytes of non-volatile configuration memory.

Configuration register is 0x2102

Log Buffer (4096 bytes):

*Aug 21 06:55:43.783: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan

*Aug 21 06:55:44.659: Unable to load pkcs11 crypto engine support for TLS

*Aug 21 06:55:45.427: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 1 (WS-X45-SUP6L-E S/N: JAE1352S7U7 Hw: 1.0) is online

*Aug 21 06:55:45.427: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 2 (WS-X4648-RJ45V+E S/N: JAE1351RHUQ Hw: 2.0) is online

*Aug 21 06:55:45.427: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 3 (WS-X4606-X2-E S/N: JAE1346NT4D Hw: 1.1) is online

000006: *Aug 21 10:55:46.383 UAE: %SYS-6-CLOCKUPDATE: System clock has been updated from 06:55:46 UTC Wed Aug 21 2013 to 10:55:46 UAE Wed Aug 21 2013, configured from console by console.

000007: *Aug 21 10:55:47.315 UAE: %SW_VLAN-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to ''.

000008: *Aug 21 10:55:47.439 UAE: %SYS-5-CONFIG_I: Configured from memory by console

000009: *Aug 21 10:55:47.660 UAE: %SYS-5-RESTART: System restarted --

Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 12.2(54)SG1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 27-Jan-11 12:19 by prod_rel_team

000010: *Aug 21 10:55:47.880 UAE: %SSH-5-ENABLED: SSH 1.99 has been enabled

000011: *Aug 21 10:55:47.880 UAE: %NTP-6-RESTART: NTP process starts

000012: *Aug 21 10:55:48.136 UAE: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.40.10.38 Port 514 started - CLI initiated

000013: *Aug 21 10:55:51.336 UAE: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.40.10.39 Port 514 started - CLI initiated

000014: *Aug 21 10:56:23.112 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=3 Port=1: Transceiver has been inserted

000015: *Aug 21 10:56:24.112 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=3 Port=2: Transceiver has been inserted

000016: *Aug 21 10:56:25.112 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=3 Port=3: Transceiver has been inserted

000017: *Aug 21 10:56:26.112 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=3 Port=4: Transceiver has been inserted

000018: *Aug 21 10:56:26.880 UAE: %NTP-6-PEERREACH: Peer 10.40.10.161 is reachable

000019: *Aug 21 10:56:27.855 UAE: %SYS-6-CLOCKUPDATE: System clock has been updated from 10:56:26 UAE Wed Aug 21 2013 to 10:56:27 UAE Wed Aug 21 2013, configured from NTP by 10.40.10.161.

000020: .Aug 21 10:56:28.087 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=1 Port=1: Transceiver has been inserted

000021: .Aug 21 10:56:30.087 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=1 Port=2: Transceiver has been inserted

000022: .Aug 21 10:56:30.087 UAE: %C4K_IOSINTF-5-TRANSCEIVERINSERTED: Slot=3 Port=5: Transceiver has been inserted

000023: .Aug 21 10:56:33.351 UAE: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.40.10.39 Port 514 started - reconnection

000024: .Aug 21 10:56:38.351 UAE: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.40.10.38 Port 514 started - reconnection

000025: Aug 21 10:57:00.855 UAE: %NTP-5-PEERSYNC: NTP synced to peer 10.40.10.161

000026: Aug 21 10:57:00.855 UAE: %NTP-6-PEERREACH: Peer 10.40.10.161 is reachable

000027: Aug 21 10:58:56.989 UAE: %AAA-3-DROPACCTFAIL: Accounting record dropped, send to server failed: system

000028: Aug 21 10:58:56.989 UAE: %SNMP-5-COLDSTART: SNMP agent on host SCHQ-DSW-02 is undergoing a cold start

000029: Aug 21 11:37:39.085 UAE: %SEC_LOGIN-1-QUIET_MODE_ON: Still timeleft for watching failures is 3 secs, [user: xxx] [Source: 10.40.10.39] [localport: 22] [Reason: Login Authentication Failed - BadPassword] [ACL: sl_def_acl] at 11:37:39 UAE Wed Aug 21 2013

000030: Aug 21 11:40:59.088 UAE: %SEC_LOGIN-5-QUIET_MODE_OFF: Quiet Mode is OFF, because block period timed out at 11:40:59 UAE Wed Aug 21 2013

NAME: "Switch System", DESCR: "Cisco Systems, Inc. WS-C4503-E 3 slot switch "

PID: WS-C4503-E , VID: V01 , SN: SPE135100LX

NAME: "Linecard(slot 1)", DESCR: "Supervisor 6L-E 10GE (X2), 1000BaseX (SFP) with 2 10GE X2 ports"

PID: WS-X45-SUP6L-E , VID: V01 , SN: JAE1352S7U7

NAME: "TenGigabitEthernet1/1", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: FNS13480S6L

NAME: "TenGigabitEthernet1/2", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: FNS13480NJU

NAME: "Linecard(slot 2)", DESCR: "10/100/1000BaseT (RJ45)+V E Series with 48 10/100/1000 baseT Premium PoE ports (Cisco/IEEE)"

PID: WS-X4648-RJ45V+E , VID: V05 , SN: JAE1351RHUQ

NAME: "Linecard(slot 3)", DESCR: "10GE (X2), 1000BaseX (SFP) with 6 10GE X2 ports"

PID: WS-X4606-X2-E , VID: V02 , SN: JAE1346NT4D

NAME: "TenGigabitEthernet3/1", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: FNS13510VMM

NAME: "TenGigabitEthernet3/2", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: FNS13510VLQ

NAME: "TenGigabitEthernet3/3", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: FNS135116KA

NAME: "TenGigabitEthernet3/4", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: AGA1344XUK9

NAME: "TenGigabitEthernet3/5", DESCR: "10Gbase-SR"

PID: X2-10GB-SR , VID: V03 , SN: AGA1344XUK8

NAME: "Fan", DESCR: "FanTray"

PID: WS-X4593-E , VID: V02 , SN: SPE135100HG

NAME: "Power Supply 1", DESCR: "Power Supply ( AC 1000W )"

PID: PWR-C45-1000AC , VID: V05 , SN: AZS13460HXM

NAME: "Power Supply 2", DESCR: "Power Supply ( AC 1000W )"

PID: PWR-C45-1000AC , VID: V05 , SN: AZS13460HX1

Please note that there was twin gig converter installed in Ten 3/6 but without SFPs.

Regards,

Amro

Eduardo Cesaro · ‎08-21-2013

Dear Amro,

You checked if your supervisor is not problems with high processing? The "fans" in power supply is operating normally?

Please check processing using the command:

sh processes cpu history

And then check if the processing is normal then check if the Power Supply have some "red light"

Let us together try to solve this

Amro BaniHani · ‎08-21-2013

Dear Eduardo,

Thank you for the rapid relpy, please find below the cpu utilization history

5555566666444445555555555555555555599999444445555544444555

100

90

80

70

60

50

40

30

20

10 ********** ************************* ***** ***

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per second (last 60 seconds)

1 1 1

6676696696696696585596696696697676996076177966716966966966

100

90

80

70

60

50

40

30

20

10 #*#**#**#**#**#**#**#**#**#*****#**#*##*#*****##*#**#**#**

0....5....1....1....2....2....3....3....4....4....5....5....

0 5 0 5 0 5 0 5 0 5

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

1111111164

1222231821

100

90

80

70

60 *

50 *

40 **

30 **

20 ***

10 ********#*

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

Leo Laohoo · ‎08-21-2013

SCHQ-DSW-02 uptime is 6 hours, 22 minutes

System returned to ROM by reload

System restarted at 10:54:47 UAE Wed Aug 21 2013

Last reload reason: Unknown reason

Sounds like you have a power problem.

Can you describe how you power up your chassis? Does this chassis have two PS active? If yes, can you elaborate if one of your two PS is on UPS or not?

Amro BaniHani · ‎08-21-2013

Dear Leo,

Thank you for the support, I have two power supplies and both of them are connected to the UPS.

Regards,

Leo Laohoo · ‎08-21-2013

Ok, that eliminates power ...

Can you check to see if your line cards are securely fastened?

Amro BaniHani · ‎08-23-2013

I cecked all the line cards, all of them are fastened tightly, this problem is bugging me now it keeps rebooting randomly, after 10 hours, 50 minutes, 3 hours, please suggest something

InayathUlla Sharieff · ‎08-22-2013

Amro,

Best link to troubleshoot high cpu on cat4500 device:

http://www.cisco.com/en/US/customer/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml

HTH

Regards

Inayath

Amro BaniHani · ‎08-23-2013

I don't believe it is a cpu issue

Leo Laohoo · ‎08-23-2013

Check your supervisor card for any sign of crashinfo file(s) and/or sub-directories.

Wait ... BOTH of your PS are connected to UPS? The SAME UPS?

Put one of your PS connected to raw power and observe.