6509 - 720 Supervisor Module Failure after Password Recovery

dminhle · ‎04-19-2015

Hi,

I attempted to perform a password recovery on a 6509 with dual WS-SUP720-3B modules running 12.2(33)SXI2A.

After changing the configuration register to 0x2142 and booting, I am getting a major error on the supervisor modules. I've changed the configuration register back to 0x2102 and am still getting the same error on both supervisor modules. I've also tried it with one module removed and one module installed.

I have about 30 seconds in where I can run commands before I get booted to rommon.

Any thoughts?

See below for the console output:

System Bootstrap, Version 8.5(3)
Copyright (c) 1994-2008 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 524288 Kbytes of main memory

Autoboot executing command: "boot bootdisk:"

Initializing ATA monitor library...

string is bootdisk:s72033-ipservicesk9-mz.122-33.SXI2a.bin

Loading image, please wait ...

Initializing ATA monitor library...

Self extracting the image... [OK]

Self decompressing the image : ################################################################################################################################################################################################### [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, s72033_sp Software (s72033_sp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:25 by prod_rel_team
Image text-base: 0x40101328, data-base: 0x4224F120

*Apr 19 05:03:02.747: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Apr 19 05:03:01.515: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor for this switch

Firmware compiled 19-May-09 12:08 by apyu Build [25600]

*Apr 19 05:03:02.747: %OIR-SP-6-CONSOLE: Changing console ownership to route processor

System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 524288 Kbytes of main memory

Download Start

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Download Completed! Booting the image.

Self decompressing the image : ############################################################################################################################################################################################################################################################################################################################ [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:00 by prod_rel_team
Image text-base: 0x40101328, data-base: 0x43A51E70

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C6509-E (R7000) processor (revision 1.5) with 458720K/65536K bytes of memory.
Processor board ID SMC133600GT
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
1 Virtual Ethernet interface
194 Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).

--- System Configuration Dialog ---

Would you like to enter the initial configuration dialog? [yes/no]: no

Press RETURN to get started!

*Apr 19 04:55:51.335: RP: Currently running ROMMON from S (Gold) region
*Apr 19 04:55:52.107: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Apr 19 04:56:00.531: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c)
Router> 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:00 by prod_rel_team
*Apr 19 05:03:03.111: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Apr 19 04:55:46.739: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Apr 19 04:55:46.747: SP: SP: Currently running ROMMON from S (Gold) region
*Apr 19 04:55:47.483: %SCHED-SP-7-WAen
Router#TCH: Attempt to set uninitialized watched boolean (address 0). -Process= "<interrupt level>", ipl= 1, pid= 3
-Traceback= 408E0820 40D74FEC 40D4B31C 40D4A1A8 40D46448 40F604B8 4173CCDC
*Apr 19 04:55:59.607: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:25 by prod_rel_team
*
Router#
Router#Apr 19 04:56:00.591: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Apr 19 04:56:00.695: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
*Apr 19 04:56:00.943: %OIR-SP-6-INSPS: Power supply inserted in slot 2
Router#
Router#
Router#
*Apr 19 04:56:08.099: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
*Apr 19 04:56:08.195: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
Router#
Router#
Router#
*Apr 19 04:56:09.567: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
Router#
Router#
Router#
Router#show diagnostic result
% Incomplete command.

Router#show diagnostic result ?
module Module Keyword

Router#show diagnostic result ,modu modjle ule 5

Current bootup diagnostic level: minimal

Module 5: Supervisor Engine 720 (Active) SerialNo : SAL1338YTZ0

Overall Diagnostic Result for Module 5 : UNTESTED
Diagnostic level at card bootup: minimal

Test results: (. = Pass, F = Fail, U = Untested)

1) TestScratchRegister -------------> U
2) TestSPRPInbandPing --------------> U
3) TestTransceiverIntegrity:

Port 1 2
----------
U U

4) TestActiveToStandbyLoopback:

Port 1 2
----------
--More-- U U

5) TestLoopback:

Port 1 2
----------
. .

6) TestNewIndexLearn ---------------> .
7) TestDontConditionalLearn --------> .
8) TestBadBpduTrap -----------------> .
9) TestMatchCapture ----------------> .
10) TestProtocolMatchChannel --------> .
11) TestFibDevices ------------------> F
12) TestIPv4FibShortcut -------------> F
13) TestL3Capture2 ------------------> F
14) TestIPv6FibShortcut -------------> F
15) TestMPLSFibShortcut -------------> F
16) TestNATFibShortcut --------------> F
17) TestAclPermit -------------------> F
18) TestAclDeny ---------------------> U
--More-- 19) TestQoSTcam ---------------------> U
20) TestL3VlanMet -------------------> U
21) TestIngressSpan -----------------> U
22) TestEgressSpan ------------------> U
23) TestNetflowInlineRewrite:

Port 1 2
----------
U U

24) TestFabricSnakeForward ----------> U
25) TestFabricSnakeBackward ---------> U
26) TestTrafficStress ---------------> U
27) TestFibTcamSSRAM ----------------> U
28) TestAsicMemory ------------------> U
29) TestAclQosTcam ------------------> U
30) TestNetflowTcam -----------------> U
31) ScheduleSwitchover --------------> U
32) TestFirmwareDiagStatus ----------> U
33) TestAsicSync --------------------> U
34) TestUnusedPortLoopback:

--More--
Router#[Ashow diagnostic result module 5

Current bootup diagnostic level: minimal

Module 5: Supervisor Engine 720 (Active) SerialNo : SAL1338YTZ0

Overall Diagnostic Result for Module 5 : UNTESTED
Diagnostic level at card bootup: minimal

Test results: (. = Pass, F = Fail, U = Untested)

1) TestScratchRegister -------------> U
2) TestSPRPInbandPing --------------> U
3) TestTransceiverIntegrity:

Port 1 2
----------
U U

4) TestActiveToStandbyLoopback:

Port 1 2
----------
--More--
Router#
*Apr 19 04:56:29.637: %DIAG-SP-3-MAJOR: M

%Software-forced reload

04:56:30 UTC Sun Apr 19 2015: Breakpoint exception, CPU signal 23, PC = 0x42735338

--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------

-Traceback= 42735338 42732E8C 423A6220 423A624C 421C1388 4223F614 4223F670 40962164 40963034 40962F54 40963D28 423259D0 42319604 42319820 42727FEC
$0 : 00000000, AT : 43FE0000, v0 : 45870000, v1 : 00000200
a0 : 46DC99C0, a1 : 44E05758, a2 : 00000000, a3 : 00000040
t0 : 00000010, t1 : 3400F101, t2 : 3400F100, t3 : FFFF00FF
t4 : 42728608, t5 : 00002F04, t6 : 80000000, t7 : 00000000
s0 : 00000000, s1 : 43EE0000, s2 : 50449158, s3 : 0000001F
s4 : 50449158, s5 : 470B3AB4, s6 : 00000000, s7 : 08A5DFC8
t8 : 08028FEC, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 43FDDE8C, sp : 5000DBB0, s8 : 00000000, ra : 42732E8C
EPC : 42735338, ErrorEPC : 870FDDF0, SREG : 3400F103
MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000000
DATA_START : 0x43A51E70
Cause 00000824 (Code 0x9): Breakpoint exception

Writing crashinfo to bootflash:crashinfo_20150419-045630

=== Flushing messages (04:56:30 UTC Sun Apr 19 2015) ===

Buffered messages:

*Apr 19 04:55:51.335: RP: Currently running ROMMON from S (Gold) region
*Apr 19 04:55:52.107: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Apr 19 04:56:00.531: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:00 by prod_rel_team
*Apr 19 05:03:03.111: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Apr 19 04:55:46.739: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*Apr 19 04:55:46.747: SP: SP: Currently running ROMMON from S (Gold) region
*Apr 19 04:55:47.483: %SCHED-SP-7-WATCH: Attempt to set uninitialized watched boolean (address 0). -Process= "<interrupt level>", ipl= 1, pid= 3
-Traceback= 408E0820 40D74FEC 40D4B31C 40D4A1A8 40D46448 40F604B8 4173CCDC
*Apr 19 04:55:59.607: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-IPSERVICESK9-M), Version 12.2(33)SXI2a, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Wed 02-Sep-09 01:25 by prod_rel_team
*Apr 19 04:56:00.591: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Apr 19 04:56:00.695: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
*Apr 19 04:56:00.943: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*Apr 19 04:56:08.099: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
*Apr 19 04:56:08.195: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
*Apr 19 04:56:09.567: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
*Apr 19 04:56:29.637: %DIAG-SP-3-MAJOR: Module 5: Online Diagnostics detected a Major Error. Please use 'show diagnostic result <target>' to see test results.
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestFibDevices failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv4FibShortcut failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestL3Capture2 failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv6FibShortcut failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
Queued messages:
*Apr 19 04:56:33.583: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
IPv6FibShortcut failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
*Apr 19 04:56:29.641: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
*Apr 19 04:56:29.971: %HA_EM-6-LOG: Mandatory.go_bootup.tcl: GOLD EEM TCL policy for boot up diagnostic
*Apr 19 04:56:30.575: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , reset by [5/0]

*** System received a Software forced crash ***

signal= 0x17, code= 0x24, context= 0x45876f54

PC = 0x42728664, SP = 0x43ed9cd8, RA = 0x413ad2f4

Cause Reg = 0x00003820, Status Reg = 0x34008002

rommon 1 >