Solved: Sup2 OS Conversion

wpalumbo06 · ‎10-07-2011

Hello,

I am embarking on the fateful journey to convert a SUP2 from CATOS to IOS and from what I have read so far, it's not a an easy process to navigate. I am ready for the challenge but will gladly accept any tips from those who have already successfully done this:-)

Thanks,

nkarpysh · ‎10-12-2011

Bill,

In crash from disk0: I see:

Active crashed three times, disabling auto-boot and dropping to rommon

This can occur when the boot variable PF_REDUN_CRASH_COUNT is

incorrectly set. This should be reset when we successfully reboot and

some past versions of code would incorrectly set this value. To fix

this, please issue the following from ROMMON:

set PF_REDUN_CRASH_COUNT=0

Then try booting again from disk0:. We should get a different sequence.

Nik

HTH,
Niko

View solution in original post

nkarpysh · ‎10-13-2011

Seems it was crashing last times due to that variable grew again due to the old crashes.

As it up please check following file systems for crashinfo files:

bootslash:

sup-bootflash:

also check:

show eobc

show ipc status

to check the current communication between rp and sp

Nik

HTH,
Niko

View solution in original post

nkarpysh · ‎10-13-2011

Ok this crash is the same one as we got in the very begining of this thread. And this was caused by PF_REDUN_CRASH_COUNT variable getting higher than 3. This is done to break continuos boot loop. So this crash does not shed light on the problems which we had booting MSFC (this one happened before control was sent to RP).

I guess physical MSFC/memory reseat cleared the issue and sub-sequent reload can be related to the variable got higher than 3 due to old failed boot attempts.

Nik

P.S.

this got to be a big thread. can you pls mark the helpfull answers through it to make it usefull for others.

Thanks,

HTH,
Niko

View solution in original post

wpalumbo06 · ‎10-11-2011

Need a little help please...

1st SUP2 converted with no problems. 2nd SUP2 failed when the switch tried to load the IOS image:

rommon 4 > boot disk0:c6k222-pk9sv-mz.122-18.SXD7b.bin

Self decompressing the image : ################################################################################################################################################################################################################################################################################### [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.

170 West Tasman Drive

San Jose, California 95134-1706

Cisco Internetwork Operating System Software

IOS (tm) c6s2p2_sp Software (c6s2p2_sp-PK9SV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 10:19 by ccai

Image text-base: 0x4002100C, data-base: 0x40D90000

00:00:03: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:03: %C6K_PLATFORM-5-MISMATCH: 'sup-bootflash' device programming algorithm mismatch detected on SP (module: 1), formatting is recommended.

00:00:03: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor

Active crashed three times, disabling auto-boot and dropping to rommon

%Software-forced reload

Unexpected exception, CPU signal 23, PC = 0x401A44D0

-Traceback= 401A44D0 401A223C 406AEFAC 406B00A8 406B1AE4 406A7BA8 406A7410 4017E15C 4017E148

$0 : 00000000, AT : 434A0000, v0 : 43FB0000, v1 : 40DE0000

a0 : 50BB64D8, a1 : 434AC1A8, a2 : 00000000, a3 : 40DB0000

t0 : 40198408, t1 : 3401FD01, t2 : 40198408, t3 : FFFF00FF

t4 : 40198408, t5 : 00000000, t6 : 0000EC62, t7 : 50C64078

s0 : 00000000, s1 : 00000000, s2 : 40DB0000, s3 : 40DC0000

s4 : 00000000, s5 : 00000000, s6 : 40C70000, s7 : 43FB0000

t8 : 00000000, t9 : 00000000, k0 : 30419001, k1 : 30410000

gp : 434AC9E4, sp : 50C64090, s8 : 00000000, ra : 401A223C

EPC : 401A44D0, ErrorEPC : 716A5567, SREG : 3401FD03

MDLO : 00000097, MDHI : 0000025C, BadVaddr : E6ADA4DB

Cause 00000024 (Code 0x9): Breakpoint exception

Writing crashinfo to bootflash:crashinfo_20111011-164834

0 packets (aggregate) dropped on throttled interfaces

=== Flushing messages (16:48:35 UTC Tue Oct 11 2011) ===

Buffered messages:

Queued messages:

00:00:05: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:04: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:05: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:05: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:05: %OIR-6-CONSOLE: Changing console ownership to switch processor

*** System received a Software forced crash ***

signal= 0x17, code= 0x24, context= 0x43fb5784

PC = 0x4019844c, Cause = 0x1020, Status Reg = 0x34018002

Thanks in advance...

Reza Sharifi · ‎10-11-2011

It seems that the IOS on the bootflash is corrupted or not loaded correctly. It is recommending a format:

detected on SP (module: 1), formatting is recommended.

Can you go back to old OS, format the flash the add the new IOS and reboot?

HTH

wpalumbo06 · ‎10-11-2011

Unfortuately I cannot format from rommon and all attempts to restore the OS have failed.

Bill

nkarpysh · ‎10-11-2011

Hi Bill,

I see you using disk0: to boot. Can you timely swap this PCMCIA disk with the 1st SUP2 which upgraded fine. For that we can be sure it have a good iamge and properly formatted card.

Try with that one - if crashed again - you will need to opene TAC acse for detailed traceback analysis of the crash.

Nik.

HTH,
Niko

wpalumbo06 · ‎10-12-2011

Looks like something is missing or corrupted -

This is what I get when I try to use the recovery procedure:

Autoboot: failed, BOOT string is empty

rommon 2 > boot

loadprog: bad file magic number: 0x0

boot: cannot load "bootflash:"

Any ideas on how to get past this point?

nkarpysh · ‎10-12-2011

Bill,

It looks that sup-bootflash: really need formatting. But you can try booting from external flash - disk0: or slot0:. Do "dev" to see your partitions under ROMMON. Then boot image ( if present) from slot0:

like

>dev

bootflash:

slot0:

***

>dir slot0:

IMAGE_NAME

> boot slot0:IMAGE_NAME

Nik

HTH,
Niko

wpalumbo06 · ‎10-12-2011

Yes,

I can boot from disk0: but it returns to ROMMON after the IOS image loads and these messages are displayed:

00:00:05: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:04: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:05: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:05: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:05: %OIR-6-CONSOLE: Changing console ownership to switch processor

*** System received a Software forced crash ***

signal= 0x17, code= 0x24, context= 0x43fb5784

PC = 0x4019844c, Cause = 0x1020, Status Reg = 0x34018002

Not sure how to get past this point??

nkarpysh · ‎10-12-2011

Bill,

In crash from disk0: I see:

Active crashed three times, disabling auto-boot and dropping to rommon

This can occur when the boot variable PF_REDUN_CRASH_COUNT is

incorrectly set. This should be reset when we successfully reboot and

some past versions of code would incorrectly set this value. To fix

this, please issue the following from ROMMON:

set PF_REDUN_CRASH_COUNT=0

Then try booting again from disk0:. We should get a different sequence.

Nik

HTH,
Niko

wpalumbo06 · ‎10-12-2011

Nik,

I got differnt results but still no luck:

rommon 1 > boot disk0:c6k222-pk9sv-mz.122-18.SXD7b.bin

Self decompressing the image : ############################################################################### ############################################################################################################## ###################################################################################### [OK]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is

subject to restrictions as set forth in subparagraph

(c) of the Commercial Computer Software - Restricted

Rights clause at FAR sec. 52.227-19 and subparagraph

(c) (1) (ii) of the Rights in Technical Data and Computer

Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.

170 West Tasman Drive

San Jose, California 95134-1706

Cisco Internetwork Operating System Software

IOS (tm) c6s2p2_sp Software (c6s2p2_sp-PK9SV-M), Version 12.2(18)SXD7b, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Fri 08-Dec-06 10:19 by ccai

Image text-base: 0x4002100C, data-base: 0x40D90000

00:00:03: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:03: %C6K_PLATFORM-5-MISMATCH: 'sup-bootflash' device programming algorithm mismatch detected on SP (modu le: 1), formatting is recommended.

00:00:03: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor

00:00:05: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

00:00:04: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.

00:00:05: %OIR-6-CONSOLE: Changing console ownership to route processor

VV'¦&'L'¤IªVªK*TK¦KSêÔTªeª5

*ÊêÍÑµ

ÕÍ¥¹¢½*¹ÍÕÉ½¹Í½±"Õ

00:03:07: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor

*** System received a Software forced crash ***

signal= 0x17, code= 0x24, context= 0x43fb5784

PC = 0x4019844c, Cause = 0x1820, Status Reg = 0x34018002

rommon 2 >

nkarpysh · ‎10-12-2011

Now we get to RP

00:00:05: %OIR-6-CONSOLE: Changing console ownership to route processor

but it can't boot and SP returns to ROMMON detecting it.

00:03:07: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor

I did not see those characters (VV'¦&'L'¤IªVªK*TK¦KSêÔTªeª5) before but seems that RP does not like the image (or formatting).

Try the following:

After SP is up and yields control to RP 
 (you see that message "Changing cnsole ownership to route processor"),
  quickly send break sequence, change config-register to 0x2102 
 and do a reset before SP detects the fact RP is not up.
 
 By that we can at least see how MSFC is booting and if what the problems it sees.
 
 Nik

HTH,
Niko

wpalumbo06 · ‎10-12-2011

Switch does not accept the break sequence. Looks like the issue is confined to the RP...there has to be a way to recover it, right?

Ravindra Simpi · ‎10-12-2011

http://www.cisco.com/en/US/ts/fn/620/fn62466.html

Check whether this field notice is applicable in this case.

nkarpysh · ‎10-12-2011

Bill,

Just want to check if this MSFC (RP) was working before when you were using CATOS on SP in hybrid mode? Just want to make sure it did not have any HW problems.

If it was then I guess we need to re-format/change the flashcards. Also you can try to boot from disk0: (PCMCIA card) from SUP which you upgarded first without no issue.

You can find bootflash locations here:

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/hardware/Config_Notes/78_12667.html

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/hardware/Config_Notes/78_14703.html

Nik,

HTH,
Niko

wpalumbo06 · ‎10-13-2011

Nik,

The SUP ran fine for a few days before I started the conversion from CATOS to IOS. I removed the 2nd SUP and converted the 1st SUP with no issues. I then shutdown the switch and removed the newly converted SUP and reinstalled SUP2. Strangely, the switch came up in rommon when I power it back up with SUP2. Since all of the initial steps to convert CATOS to IOS were complete, I just started the conversion process from rommon which I believe is somewhere around step 14 or 15. It looks like the SP is loading the IOS image correctly but failing when it tries to boot the MSFC. Are there any other steps that I can try to correct this?

Thanks,