6509 crash when upgrading supervisor card.

[phil.g.james]

Can anyone help with a problem I have encountered when upgrading a Cisco 6509 chassis with a new supervisor card from Sup2 to Sup 720b.

The 6509 loads and then crashes completely, and when rebooted reloads into rommon. The same upgrade was performed on a similar switch with no apparent problems.

The following error messages have been picked up in the log.

- RELIABILITY DRIVER: wrong signature on NVFLASH

Spurious interrupts: 4

System was restarted by error - a Software forced crash, PC 0x42738778 at 16:36:56 GMT Sat Nov 19 2011

%C6K_PLATFORM-2-PEER_RESET: RP is being reset by the SP

%Software-forced reload

Does anyone have any idea what might be causing this problem. The SP and RP IOS is the same. And all of the cards in the 6509 should be supported by the supervisor.

[Amit Singh]

I have done some internal research and this "reliability driver" just seems to a message that was programed in older IOS images for NVRAM signatures loaded by the manufacturing. This has been removed after the 12.17SX train. This is just a cosmetic thing and you either igrnore the message or upgrade to a later IOS.

What is the IOS you are running?

Please could you paste the booting sequence of the switch. It looks like there is some other issue with the switch.

Cheers,

-amit singh

[phil.g.james]

Thanks Amit

I have pasted the booting sequence below, unfortunatly there was not much time for diagnostics, as the change window we had was coming to a close. The first load you can see captured here is from the rommon reboot and it has picked up an older version of the IOS from the sup-bootflash, but the second reload has the correct software.

Loading image, please wait ...

Self decompressing the image :

#################################################################################################################################################################################

############################################### [OK]
              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco Internetwork Operating System Software
IOS (tm) s72033_sp Software (s72033_sp-SP-M), Version 12.2(17d)SXB7, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Thu 24-Feb-05 21:20 by kellmill
Image text-base: 0x40020FBC, data-base: 0x40DC0000

00:00:03: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor

00:00:07: %OIR-SP-6-CONSOLE: Changing console ownership to route processor

System Bo otstrap, Version 12.2(17r)S2, RELEASE SOFTWARE (fc1)
TAC Sup port: ht tp:/ /w w w.cisco.com/tac
Copyright (c) 20 04 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 524288 Kbytes of main memory

Download Start
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
Download

Completed! Booting the image.
Self decompressing the image :

########################################################################################################################################## [OK]
              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-PS-M), Version 12.2(17d)SXB7, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Thu 24-Feb-05 21:16 by kellmill
Image text-base: 0x40020FBC, data-base: 0x41D38000

cisco WS-C6509 (R7000) processor (revision 3.0) with 458752K/65536K bytes of memory.
Processor board ID TBP06190154
SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
Last reset from power-on
X.25 software, Version 3.0.0.
Bridging software.
1 Virtual Ethernet/IEEE 802.3  interface(s)
144 FastEthernet/IEEE 802.3 interface(s)
2 Gigabit Ethernet/IEEE 802.3 interface(s)
1917K bytes of non-volatile configuration memory.
8192K bytes of packet buffer memory.

65536K bytes of Flash internal SIMM (Sector size 512K).
upgrade fpd auto
^
% Invalid input detected at '^' marker.

boot-start-marker
    ^
% Invalid input detected at '^' marker.

boot-end-marker
    ^
***********************************************************************************************************************************

%SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) s72033_rp Software (s72033_rp-PS-M), Version 12.2(17d)SXB7, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Thu 24-Feb-05 21:16 by kellmill
00:00:58: curr is 0x0

00:00:58: RP: Currently running ROMMON from S (Gold) region
%SNMP-5-COLDSTART: SNMP agent on host MISNLCYLSW01B is undergoing a cold start

Press RETURN to get started!

00:00:07: %OIR-SP-6-CONSOLE: Changing console ownership to route processor

Firmware compiled 02-Feb-05 16:48 by integ Build [100]

- RELIABILITY DRIVER: wrong signature on NVFLASH

00:00:56: SP: SP: Currently running ROMMON from S (Gold) region
00:01:12: %SYS-SP-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) s72033_sp Software (s72033_sp-SP-M), Version 12.2(17d)SXB7, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Thu 24-Feb-05 21:20 by kellmill

********************************************************************************************************************************

%SYS-5-RESTART: System restarted --
Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 27-Oct-09 11:12 by prod_rel_team
*Nov 19 15:18:48.035: RP: Currently running ROMMON from S (Gold) region
%SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
*Nov 19 15:18:43.523: SP: SP: Currently running ROMMON from S (Gold) region
%C6K_PLATFORM-SP-4-CONFREG_BREAK_ENABLED: The default factory setting for config register is 0x2102.It is advisable to retain 1 in 0x2102 as it prevents returning to ROMMON when

break is issued.
%SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, s72033_sp Software (s72033_sp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 27-Oct-09 11:34 by prod_rel_team
%SNMP-5-COLDSTART: SNMP agent on host MISNLCYLSW01B is undergoing a cold start
%MFIB_CONST_RP-6-REPLICATION_MODE_CHANGE: Replication Mode Change Detected. Current system replication mode is Ingress
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
%HSRP-5-STATECHANGE: Vlan2 Grp 10 state Standby -> Active
%HSRP-5-STATECHANGE: Vlan800 Grp 80 state Listen -> Active
%HSRP-5-STATECHANGE: Vlan5 Grp 13 state Standby -> Active
%HSRP-5-STATECHANGE: Vlan162 Grp 162 state Standby -> Active
%HSRP-5-STATECHANGE: Vlan3 Grp 11 state Standby -> Active
%HSRP-5-STATECHANGE: Vlan300 Grp 14 state Speak -> Standby
%HSRP-5-STATECHANGE: Vlan101 Grp 101 state Standby -> Active
%HSRP-5-STATECHANGE: Vlan161 Grp 161 state Standby -> Active
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
%HSRP-5-STATECHANGE: Vlan4 Grp 12 state Standby -> Active
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
CMD: 'en' 16:35:17 GMT Sat Nov 19 2011
CMD: 'sh int status' 16:35:27 GMT Sat Nov 19 2011
CMD: 'sh int status mod 6' 16:35:38 GMT Sat Nov 19 2011
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
CMD: 'sh run int g6/1' 16:36:09 GMT Sat Nov 19 2011
CMD: 'sh int trunk' 16:36:42 GMT Sat Nov 19 2011
CMD: 'sh trunk' 16:36:46 GMT Sat Nov 19 2011
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).
CMD: 'sh ip ro' 16:36:53 GMT Sat Nov 19 2011
%C6K_PLATFORM-2-PEER_RESET: RP is being reset by the SP

%Software-forced reload

16:36:56 GMT Sat Nov 19 2011: Breakpoint exception, CPU signal 23, PC = 0x42738778

--------------------------------------------------------------------
   Possible software fault. Upon reccurence, please collect
   crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------

-Traceback= 42738778 427362CC 41F9AFCC 41F9C650 4231CA10 4231CB68 4272B95C
$0 : 00000000, AT : 43FF0000, v0 : 45880000, v1 : 00000000
a0 : 50EA96A4, a1 : 00008100, a2 : 00000000, a3 : 00000000
t0 : 4272BA48, t1 : 34008101, t2 : 4272BA70, t3 : FFFF00FF
t4 : 4272BA48, t5 : 500122E8, t6 : 00000000, t7 : 0000A691
s0 : 00000000, s1 : 43EE0000, s2 : 00000000, s3 : 43E60000
s4 : 43E60000, s5 : 43E60000, s6 : 436D0000, s7 : 00000001
t8 : 5001234C, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 43FE9414, sp : 50012418, s8 : 436D0000, ra : 427362CC
EPC  : 42738778, ErrorEPC : BB3DBB7F, SREG     : 34008103
MDLO : 00000000, MDHI     : 00000000, BadVaddr : 00000000
DATA_START : 0x43A5C5D0
Cause 00000824 (Code 0x9): Breakpoint exception

========= Start of Crashinfo Collection (16:36:56 GMT Sat Nov 19 2011) =========
For image:
Cisco IOS Software, s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 27-Oct-09 11:12 by prod_rel_team

========= Show Alignment =======================================================

No alignment data has been recorded.

No spurious memory references have been recorded.

========= Malloc and Free Traces ===============================================

MallocFree Trace: ixmallocfree=0x1A  ptr=0x44FD9580
44FD94B0: 46C26FE8 4041DAF4 46C27030 60000022 48DCCA08 30000052 48DCCA08 413A2F50
44FD94D0: 48DCCA08 413A34B8 52659648 40BF98F4 52659718 6000002C 52659648 40000054
44FD94F0: 52659648 40BF961C 52659648 40BF98F4 52659718 6000002C 519E077C 4056CE40
44FD9510: 519E053C 4056CE48 519DFF9C 4056CE40 47D580D4 4056CE48 468750D8 4056CE40
44FD9530: 50B73E64 4056CE48 48DCCF68 400003FC 48DCCF68 40CFAB20 48DCCF68 40CFAC30
44FD9550: 48DCD788 6191943C 52659648 40000054 52659648 40BF961C 52659648 40BF98F4
44FD9570: 52659718 6000002C 52659648 40000054 52659648 40BF961C 52988624 4220AED4
44FD9590: 52988890 66B37BB8 48DCCF68 421F9018 48DCDFAC 6191902A 52988624 4000080E
44FD95B0: 52988624 421F8F30 48DCCF68 40000122 48DCCF68 422094BC 48DCCF68 4220AED4
44FD95D0: 48DCD1D4 61919716 52988624 421F9018 52989668 66B374CC 48B25514 4041D1F8
44FD95F0: 46C26EBC 4041EB8C 46C26F10 60000016 5257F90C 40000010 5257F90C 4041DC30
44FD9610: 528C0DDC 4041DF08 528C0E24 60000016 528C0DDC 40000016 528C0DDC 4041EAF0
44FD9630: 46C26EBC 40000010 46C26EBC 4041D898 528C21D8 4041DAF4 528C2220 6000001A
44FD9650: 48B25514 4041D218 48A22968 4041D218 48A22834 4041D1F8 46C27C28 4041EB8C
44FD9670: 528C21D8 40000010 528C21D8 4041DC30 48B25648 4041DF08 48B25690 60000016
44FD9690: 48B25648 40000016 48B25648 4041EAF0 5257FC60 40000010 5257FC60 4041D898

========= Chunk Malloc and Chunk Free Traces=====================

44FD90B0: 4688598C 40931384 4092F6C8 4688598C 40931310 4092F6C8
44FD9098: 4688598C 40931384 4092F6C8 50CC5200 4092A744 4092919C
44FD9080: 4688598C 40931310 4092F6C8 4688598C 40931384 4092F6C8
44FD9068: 4688598C 40931310 4092F6C8 51576C24 41D41B6C 41D41974
44FD9050: 51576C24 41D41AE4 41D41974 51576C24 41D41B6C 41D41974
44FD9038: 51576C24 41D41AE4 41D41974 524C7E0C 413A3D80 413A5408
44FD9020: 52595FE0 413A293C 413A548C 52595FE0 413A27C0 413A548C
44FD9008: 524C7E0C 413A2AB8 413A5408 46FB3C78 40CAC03C 4076E44C
44FD8FF0: 46FB3EF8 40CABF98 4076E44C 5246ED8C 4060DA68 4060DAB0
44FD8FD8: 5246ED8C 4060D9B8 4060DAB0 51689500 40C8D49C 40C8EA3C
44FD8FC0: 516895F0 40C8D49C 40C8EA3C 524C7E0C 413A3D80 413A5408
44FD8FA8: 52595FE0 413A293C 413A548C 52595FE0 413A27C0 413A548C
44FD8F90: 524C7E0C 413A2AB8 413A5408 51576C24 41D41B6C 41D41974
44FD8F78: 51576C24 41D41AE4 41D41974 46FB3EF8 40CAC03C 4076E44C
44FD8F60: 46FB3D68 40CABF98 4076E44C 46FB3D68 40CAC03C 4076E44C
44FD8F48: 46FB3F48 40CABF98 4076E44C 51689488 4074E7B8 40C8EA3C
44FD8F30: 51576C24 41D41B6C 41D41974 51576C24 41D41AE4 41D41974
44FD8F18: 5246ED8C 4060DA68 4060DAB0 5246ED8C 4060D9B8 4060DAB0
44FD8F00: 524C7E0C 413A3D80 413A5
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet4/32 (not half duplex), with MISNLCYXSANSW02(FOX1416GAE4) mgmt0 (half duplex).408 524C7E0C 413A2AB8 413A5408
44FD8EE8: 45FBAE44 405349A4 4052DF68 502B9FF4 405349A4 4052DFEC
44FD8ED0: 502BC0DC 405349A4 4052DFEC 502BA744 405349A4 4052DFEC
44FD8EB8: 502BC0DC 405348C4 4052DFEC 502BC0DC 405349A4 4052DFEC
44FD8EA0: 502BA744 405348C4 4052DFEC 502BA744 405349A4 4052DFEC
44FD8E88: 502BC0DC 405348C4 4052DFEC 502BC0DC 405349A4 4052DFEC
44FD8E70: 502BA744 405348C4 4052DFEC 502BA744 405349A4 4052DFEC
44FD8E58: 502BC0DC 405348C4 4052DFEC
44FD914C: 502B9FF4 405348C4 4052DFEC 502B9FF4 405349A4 4052DFEC
44FD9134: 502BC0DC 405349A4 4052DFEC 502B9FF4 405348C4 4052DFEC
44FD911C: 502B9FF4 405349A4 4052DFEC 502BB23C 405349A4 4052DFEC
44FD9104: 502B9FF4 405348C4 4052DFEC 502B9FF4 405349A4 4052DFEC
44FD90EC: 502BB23C 405348C4 4052DFEC 502BC0DC 405348C4 4052DFEC
44FD90D4: 502BC0DC 405349A4 4052DFEC 502BB23C 405349A4 4052DFEC
44FD90BC: 502BC0DC 405348C4 4052DFEC

========= Stack Trace ==========================================================

-Traceback= 42738778 427362CC 41F9AFCC 41F9C650 4231CA10 4231CB68 4272B95C

========= Context ==============================================================

s72033_rp Software (s72033_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXI3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Compiled Tue 27-Oct-09 11:12 by prod_rel_team
Signal = 23, Code = 0x24, Uptime 00:07:06
$0 : 00000000, AT : 43FF0000, v0 : 45880000, v1 : 00000000
a0 : 50EA96A4, a1 : 00008100, a2 : 00000000, a3 : 00000000
t0 : 4272BA48, t1 : 34008101, t2 : 4272BA70, t3 : FFFF00FF
t4 : 4272BA48, t5 : 500122E8, t6 : 00000000, t7 : 0000A691
s0 : 00000000, s1 : 43EE0000, s2 : 00000000, s3 : 43E60000
s4 : 43E60000, s5 : 43E60000, s6 : 436D0000, s7 : 00000001
t8 : 5001234C, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 43FE9414, sp : 50012418, s8 : 436D0000, ra : 427362CC

[Amit Singh]

Hi Phil,

There are two booting sequence in the outputs you have pasted.

1. In the first sequence you have the system booting up with Version 12.2(17d)SXB7. That's why you see that error in the system "- RELIABILITY DRIVER: wrong signature on NVFLASH".  At the end of the output it really did not throw any error so its very hard to find why system restarted and went on to boot with the second IOS. As I mentioned that it was a cosmetic thing in the old IOS related to the NVRAM programming, you could ignore that error.

2. In the second part of the booting sequence, the system booted up with Version 12.2(33)SXI3. The system tried to boot up fine, however towards the end SP resetted the RP and it detected some software issue with the system.

--------------------------------------------------------------------

   Possible software fault. Upon reccurence, please collect

   crashinfo, "show tech" and contact Cisco Technical Support.

--------------------------------------------------------------------

It looks loks like there is issue with the IOS Version 12.2(33)SXI3 that you are running on the system. I tried to put the traceback error in Cisco's output decoder tool but could not get a concerete output/answer.

How is boot variable configured to boot the system? What happens when you set the boot variable to 12.2(17d)SXB7

only, does the system boot up fine?

Can you try loading a new IOS and see if it resolves the issue?

HTH,

-amit singh

[phil.g.james]

Thanks for the reply Amit, I have managed to find a spare 6509 chassis to test the sup card with as the above one is a live service, will try out a few things next week when I get to the spare chassis.

Phil