Catalyst 6506-e stays on rommon due "Possible software faulty" and problem on SUP720

[joleloves12]

My 2 Catalyst 6506-E go on Rommon and boot up any Ios image. This is the crashinfo collected:

cisco WS-C6506-E (R7000) processor (revision 1.2) with 458720K/65536K bytes of memory.
Processor board ID SAL1428MUNZ
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from s/w reset
1 Virtual Ethernet interface
50 Gigabit Ethernet interfaces
16 Ten Gigabit Ethernet interfaces
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
Missing file name
Logging of %SNMP-3-AUTHFAIL is enabled
Warning: sup-bootdisk:system does not exist.  Command retained.


Press RETURN to get started!


*May 31 00:14:51.119: % SNMP ID Persistence Error : Unable to open file : No such file or directory
*May 31 00:14:53.055: RP: Currently running ROMMON from S (Gold) region
*May 31 00:14:53.887: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
000004: *May 31 00:15:00.439 UTC: %SYS-5-CONFIG_I: Configured from memory by console
000005: *May 31 00:15:03.847 UTC: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI14, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 04-Sep-14 00:38 by prod_rel_team
000006: *May 31 00:15:03.891 UTC: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.10.140.5 port 514 started - CLI initiated
000007: *May 31 00:15:06.219 UTC: %SNMP-5-COLDSTART: SNMP agent on host DIST-6500-6E is undergoing a cold start
000008: *May 31 00:16:19.427: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
000009: *May 31 00:14:48.467: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
*May 31 00:14:48.483: SP: SP: Currently running ROMMON from S (Gold) region
000010: *May 31 00:14:49.223: %SCHED-SP-7-WATCH: Attempt to set uninitialized watched boolean (address 0). -Process= "<interrupt level>", ipl= 1, pid= 3
-Traceback= 408EF7E4 40DC31EC 40DC3218 40D98EE0 40D97D6C 40D94000 40FB405C 417B9B3C
000011: *May 31 00:15:02.867: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI14, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 04-Sep-14 00:58 by prod_rel_team
000012: *May 31 00:15:03.971 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 1
000013: *May 31 00:15:04.079 UTC: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
000014: *May 31 00:15:04.327 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 2
000015: *May 31 00:15:04.431 UTC: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
000016: *May 31 00:15:04.535 UTC: %C6KPWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
000017: *May 31 00:15:07.990 UTC: %C6KENV-SP-4-FANHPMODE: Fan-tray 1 is operating in high power mode
000018: *May 31 00:15:15.157 UTC: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
000019: *May 31 00:15:15.257 UTC: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
000020: *May 31 00:15:16.710 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...

%Software-forced reload

 Early Notification of crash condition..

 00:15:30 UTC Thu May 31 2018: Breakpoint exception, CPU signal 23, PC = 0x42DB3234


--------------------------------------------------------------------
   Possible software fault. Upon reccurence, please collect
   crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------

-Traceback= 42DB3234 42DB0D74 42A103A8 42A103D4 4281FC24 4289FE80 4289FEDC 40993B48 40994A18 40994938 4099570C 4298AA48 4297C4AC 4297C6C8 42DA58E4
$0 : 00000000, AT : 44A60000, v0 : 442B0000, v1 : 00000000
a0 : 50C680CC, a1 : 0000F100, a2 : 00000000, a3 : 00000000
t0 : 00000020, t1 : 3400F101, t2 : 3400C100, t3 : FFFF00FF
t4 : 42DA60C0, t5 : 656D732C, t6 : 2E0A436F, t7 : 65642054
s0 : 00000000, s1 : 44950000, s2 : 504766C0, s3 : 0000001F
s4 : 504766C0, s5 : 5104FC98, s6 : 00000000, s7 : 08B177C8
t8 : 08028FEC, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 44A63C0C, sp : 5000DBB0, s8 : 00000000, ra : 42DB0D74
EPC  : 42DB3234, ErrorEPC : CBBFD491, SREG     : 3400F103
MDLO : 00000000, MDHI     : 00000000, BadVaddr : 00000000
DATA_START : 0x4441B9D0
Cause 00000824 (Code 0x9): Breakpoint exception

Writing crashinfo to bootflash:crashinfo_20180531-001530-UTC

=== Flushing messages (00:15:30 UTC Thu May 31 2018) ===

Buffered messages:

*May 31 00:14:53.887: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
000004: *May 31 00:15:00.439 UTC: %SYS-5-CONFIG_I: Configured from memory by console
000005: *May 31 00:15:03.847 UTC: %SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M), Version 12.2(33)SXI14, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Thu 04-Sep-14 00:38 by prod_rel_team
000007: *May 31 00:15:06.219 UTC: %SNMP-5-COLDSTART: SNMP agent on host DIST-6500-6E is undergoing a cold start
000008: *May 31 00:16:19.427: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
000009: *May 31 00:14:48.467: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan. The Bridge IDs of all active STP instances have been updated, which might change the spanning tree topology
000011: *May 31 00:15:02.867: %SYS-SP-5-RESTART: System restarted --
000013: *May 31 00:15:04.079 UTC: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
000015: *May 31 00:15:04.431 UTC: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
000016: *May 31 00:15:04.535 UTC: %C6KPWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
000017: *May 31 00:15:07.990 UTC: %C6KENV-SP-4-FANHPMODE: Fan-tray 1 is operating in high power mode
000018: *May 31 00:15:15.157 UTC: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
000019: *May 31 00:15:15.257 UTC: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
000021: *May 31 00:15:29.395 UTC: %DIAG-SP-3-MAJOR: Module 5: Online Diagnostics detected a Major Error. Please use 'show diagnostic result <target>' to see test results.
000022: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestFibDevices failed
000023: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv4FibShortcut failed
000024: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv6FibShortcut failed
000025: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
000026: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
Queued messages:
000028: *May 31 00:15:30.023 UTC: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

000021: *May 31 00:15:29.395 UTC: %DIAG-SP-3-MAJOR: Module 5: Online Diagnostics detected a Major Error. Please use 'show diagnostic result <target>' to see test results.
000022: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestFibDevices failed
000023: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv4FibShortcut failed
000024: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv6FibShortcut failed
000025: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
000026: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
000027: *May 31 00:15:29.755 UTC: %HA_EM-6-LOG: Mandatory.go_bootup.tcl: GOLD EEM TCL policy for  boot up diagnostic
000028: *May 31 00:15:30.003 UTC: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , reset by [5/0]
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46639064
  PC = 0x42da611c, SP = 0x44948868, RA = 0x413eaef0
  Cause Reg = 0x00003820, Status Reg = 0x34008002
rommon 1 > dir bootflash:

This operation is not permitted after send-break.
rommon 2 > boot
Please reset before booting
rommon 3 > reset

System Bootstrap, Version 12.2(17r)SX6, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 524288 Kbytes of main memory

System Bootstrap, Version 8.5(3)
Copyright (c) 1994-2008 by cisco Systems, Inc.

Testing lower main memory - data equals address
Testing lower main memory - checkerboard
Testing lower main memory - inverse checkerboard
Clearing lower memory for cache initialization
Clearing bss
Clearing autoboot state machine
melody_present_reg: 1st read w/ 0x5555
melody_present_reg: 2nd read w/ 0xaaaa, reversed: 0x5555
Bootdisk adapter is detected, enabling bootdisk access...
Reprogramming CS1 w/ Melody value...

Reading monitor variables from NVRAM
Reset reason for CPU board 0xffff , BaseBoard 0x200ffff, display 0x0System Reset by Power On.

Enabling interrupts
Initializing TLB
Initializing cache
Initializing required TLB entries
Initializing main memory
Sizing NVRAM
Initializing PCMCIA controller
Exiting init
Cat6k-Sup720/SP processor with 524288 Kbytes of main memory

rommon 1 >

[Leo Laohoo]

---

@joleloves12 wrote:

000022: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestFibDevices failed
000023: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv4FibShortcut failed
000024: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv6FibShortcut failed
000025: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
000026: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
000027: *May 31 00:15:29.755 UTC: %HA_EM-6-LOG: Mandatory.go_bootup.tcl: GOLD EEM TCL policy for  boot up diagnostic
000028: *May 31 00:15:30.003 UTC: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , reset by [5/0]
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46639064
  PC = 0x42da611c, SP = 0x44948868, RA = 0x413eaef0
  Cause Reg = 0x00003820, Status Reg = 0x34008002

---

Re-seat the supervisor card and try again.  If the same fault appears, raise a TAC Case.  At the end of the day, an IOS upgrade maybe required.

[joleloves12]

Have reseated the module, even changed the position into the chassis. Result, the same error message notifying that a major error on Supervisor. This is the message:

65536K bytes of Flash internal SIMM (Sector size 512K).
Missing file name
Logging of %SNMP-3-AUTHFAIL is enabled
Warning: sup-bootdisk:system does not exist.  Command retained.

Compiled Thu 04-Sep-14 00:58 by prod_rel_team

000011: *May 31 00:15:02.867: %SYS-SP-5-RESTART: System restarted --
000012: *May 31 00:15:03.971 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 1
000013: *May 31 00:15:04.079 UTC: %C6KPWR-SP-4-PSOK: power supply 1 turned on.
000014: *May 31 00:15:04.327 UTC: %OIR-SP-6-INSPS: Power supply inserted in slot 2
000015: *May 31 00:15:04.431 UTC: %C6KPWR-SP-4-PSOK: power supply 2 turned on.
000016: *May 31 00:15:04.535 UTC: %C6KPWR-SP-4-PSREDUNDANTBOTHSUPPLY: in power-redundancy mode, system is operating on both power supplies.
000017: *May 31 00:15:07.990 UTC: %C6KENV-SP-4-FANHPMODE: Fan-tray 1 is operating in high power mode
000018: *May 31 00:15:15.157 UTC: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
000019: *May 31 00:15:15.257 UTC: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
000020: *May 31 00:15:16.710 UTC: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...

%Software-forced reload

000021: *May 31 00:15:29.395 UTC: %DIAG-SP-3-MAJOR: Module 5: Online Diagnostics detected a Major Error. Please use 'show diagnostic result <target>' to see test results.
000022: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestFibDevices failed
000023: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv4FibShortcut failed
000024: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestIPv6FibShortcut failed
000025: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestMPLSFibShortcut failed
000026: *May 31 00:15:29.399 UTC: %CONST_DIAG-SP-3-BOOTUP_TEST_FAIL: Module 5: TestNATFibShortcut failed
000027: *May 31 00:15:29.755 UTC: %HA_EM-6-LOG: Mandatory.go_bootup.tcl: GOLD EEM TCL policy for  boot up diagnostic
000028: *May 31 00:15:30.003 UTC: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , reset by [5/0]
*** System received a Software forced crash ***
signal= 0x17, code= 0x24, context= 0x46639064
  PC = 0x42da611c, SP = 0x44948868, RA = 0x413eaef0
  Cause Reg = 0x00003820, Status Reg = 0x34008002
rommon 1 >

[Leo Laohoo]

Contact TAC for assistance.
I believe this could be hardware related issue.