Downgrade WS-C3850-24T-E (16.x -> 3.x) -

[christoffer]

I'm trying to perform a downgrade on this model from IOS XE Denali 16.2.1 to 3.6.4E but receive an "error" when rebooting the switch.

"A system init job [rc3-local] exited with the result [failed]."

The switch is currently running in install mode and I have tried to downgrade to 16.1.2 and from there go to 3.6.4E, same error.

All packages are Digitally Signed
Starting System Services
**************************************************************
Warning!
A system init job [rc3-local] exited with the result [failed].
The system maybe unstable because of this.
**************************************************************

After this I have to do an emergency-install with an 16.x image to get it up and running again.

Has anyone experienced this, and maybe have a solution to my problem?

Thanks in advance,

/Christoffer

[Furose M]

Hi Chris,

Does the switch not booting at all in 3.6.4 version after this error?

can you try booting 3.6.4 from usb via .bin file and then format flash: and then do "software expand" in 3.6.4 to flash:?

[christoffer]

[toc:faq]

No, not booting at all until I install a 16.x version again.

Same thing when I try an emergency-install to version 3.6.4:

switch: emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

The bootflash will be erased during install operation, continue (y/n)?y

Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin)...

Reading full image into memory........................done

Bundle Image

--------------------------------------

Kernel Address    : 0x534305d4

Kernel Size       : 0x3faba9/4172713

Initramfs Address : 0x5382b180

Initramfs Size    : 0xdf2d3b/14626107

Compression Format: .mzip

Bootable image at @ ram:0x534305d4

Bootable image segment 0 address range [0x81100000, 0x820a0000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x816733a0

Loading Linux kernel with entry point 0x816733a0 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

Downloading bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin...

Validating bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin...

Installing bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin...

Verifying bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin...

Package cat3k_caa-base.SPA.03.06.04.E.pkg is Digitally Signed

Package cat3k_caa-drivers.SPA.03.06.04.E.pkg is Digitally Signed

Package cat3k_caa-infra.SPA.03.06.04.E.pkg is Digitally Signed

Package cat3k_caa-iosd-universalk9.SPA.152-2.E4.pkg is Digitally Signed

Package cat3k_caa-platform.SPA.03.06.04.E.pkg is Digitally Signed

Package cat3k_caa-wcm.SPA.10.2.140.0.pkg is Digitally Signed

Preparing flash...

Syncing device...

Emergency Install successful... Rebooting

Restarting system.

Booting...

Getting rest of image

Reading full image into memory...Check base package header ...: done = 16384

Getting rest of image

Reading full image into memory....done

Reading full base package into memory...: done = 82665136

Bundle Image

--------------------------------------

Kernel Address    : 0x5342f350

Kernel Size       : 0x418a9b/4295323

Initramfs Address : 0x53847dec

Initramfs Size    : 0xe780c3/15171779

Compression Format: .mzip

Bootable image at @ ram:0x5342f350

Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377

Loading Linux kernel with entry point 0x8166a600 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed

Starting System Services

**************************************************************

                           Warning!           

A system init job [rc3-local] exited with the result [failed].

The system maybe unstable because of this.            

**************************************************************

FIPS(NGWC): Flash Key Check : Begin

FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled

Tried to boot cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin with boot system command after performing an emergency-install again to 16.1.1. Same outcome :(

Switch# conf t

Switch(config)# boot system switch all flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

Switch# Copy run start

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin;

Boot Variables on next reload:

BOOT variable = flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin;

Manual Boot = no

Enable Break = no

Switch#reload

Reload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

*May 24 06:10:22.691: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.

Chassis 1 reloading, reason - Reload command

May 24 06:10:33.874 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested 

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!

May 24 06:10:52.480 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: process exit with reload stack code

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!

reboot: Restarting system

Booting...Reading full image into memory....................................................................................................................................................................................................................................................................................................done

Bundle Image

--------------------------------------

Kernel Address : 0x5342f294

Kernel Size : 0x418a9b/4295323

Initramfs Address : 0x53847d30

Initramfs Size : 0xe780c3/15171779

Compression Format: .mzip

Bootable image at @ ram:0x5342f294

Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin" uncompressed and installed, entry point: 0x8166a600

Loading Linux kernel with entry point 0x8166a600 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed

Starting System Services

**************************************************************

Warning! 

A system init job [rc3-local] exited with the result [failed].

The system maybe unstable because of this. 

**************************************************************

FIPS(NGWC): Flash Key Check : Begin

FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled

---- NOTHING HAPPENS ----

Release notes

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-2/release_notes/ol-16-2-3850.html#pgfId-1195184

Says the following:

When you downgrade to a Cisco IOS XE 3.xE image, your boot loader will not automatically downgrade. It will remain updated. The new boot loader can support booting both Cisco IOS XE 3.xE releases as well as Cisco IOS XE Denali16.x releases.

Don´t know what else to try at this point. Any advice appreciated! :)

[Furose M]

can you try formatting the flash once and try booting the bin file from USB please?

[christoffer]

Same result as before :/

[Furose M]

from what i remember, the ucode driver is seperately read from flash in 3.6.4, but in 3.7.3, it is integrated in IOS.

i expected everything to be wiped after formatting flash/ anyhow...

can you try upgrading to 3.7.3 version? we shouldnt be seeing any driver related issues in 3.7.3.

this will be my last bet!!

[christoffer]

Thanks for that information. An installation to 3.7.3 works fine, but same result again when I try to jump to 3.6.4 from there.

Switch# software install file flash:/cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin  new

Preparing install operation ...

[1]: Starting install operation

[1]: Expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

[1]: Copying package files

[1]: Package files copied

[1]: Finished expanding bundle flash:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

[1]: Verifying and copying expanded package files to flash:

[1]: Verified and copied expanded package files to flash:

[1]: Starting compatibility checks

[1]: % Candidate package compatibility checks failed because the following

       package dependencies were not satisfied. Operation aborted.

       Package cat3k_caa-platform requires cat3k_caa-drivers package version >=3.6.1, but the

       candidate package set includes cat3k_caa-drivers package version 03.06.00.PSR2.73

[1]: % An internal error was encountered. Operation aborted.

switch: dir usbflash0:

Directory of usbflash0:/

   141  -rw-  302988468  cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin

Reading full image into memory....................................................................................................................................................................................................................................................................................................done

Bundle Image

--------------------------------------

Kernel Address    : 0x5344e294

Kernel Size       : 0x418a9b/4295323

Initramfs Address : 0x53866d30

Initramfs Size    : 0xe780c3/15171779

Compression Format: .mzip

Bootable image at @ ram:0x5344e294

Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "usbflash0:cat3k_caa-universalk9.SPA.03.06.04.E.152-2.E4.bin" uncompressed and installed, entry point: 0x8166a600

Loading Linux kernel with entry point 0x8166a600 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed

Starting System Services

**************************************************************

                           Warning!           

A system init job [rc3-local] exited with the result [failed].

The system maybe unstable because of this.            

**************************************************************

FIPS(NGWC): Flash Key Check : Begin

FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled

Do you have any more tips on what I could try based on this message?

       Package cat3k_caa-platform requires cat3k_caa-drivers package version >=3.6.1, but the

       candidate package set includes cat3k_caa-drivers package version 03.06.00.PSR2.73

Thanks in advance,

/Christoffer

[Furose M]

you could try renaming/moving this file and see if it helps.

cat3k_caa-drivers.SSA.03.06.00.PSR2.73.pkg

[christoffer]

I may be stupid, but sda8 is read only for me. Can you bypass this somehow?

Also just tried to jump from 3.7.0 to 3.6.4 with the same result as before.

Is there somehow a way to download cat3k_caa-drivers package version >=3.6.1 and insert it manually into sda8 ?

[Bhishma Khanna]

Hi,

Apart from what Furose suggested, we can try moving an old package from sda8: folder to /: 

Here are the steps:

#conf t

# boot manual

#exit

#wr

# reload

You would enter the Switch: prompt

run the commands as:

Switch: dir Sda8: 

check if you are able to see package file cat3k_caa-drivers.SSA.0.DEV-0.pkg

If yes then move it to another location by:

mv cat3k_caa-drivers.SSA.0.DEV-0.pkg /

Then try to boot with the 3.6.4E image.

Thanks,

Bhishma Khanna

[christoffer]

On sda8, these are the files I have:

switch: dir sda8:
Directory of sda8:/

2 drwx 1024 .
2 drwx 1024 ..
12 -rw- 8352456 cat3k_caa-drivers.SSA.03.06.00.PSR2.73.pkg
5977 -rw- 6769352 cat3k_caa-drivers.SSA.03.02.03.XSR1.61.pkg
5978 -rw- 10537672 cat3k_caa-drivers.SSA.03.03.04.ZSR2.03.pkg

Should I try move all these?

[Bhishma Khanna]

Hi,

No, we do not want to change the location of any of these package files. In this case I think you should not go with 3.6.X train unless it's not a very specific requirement.

Thanks,

Bhishma Khanna

[mattmelick1]

What was the final outcome of this problem. I'm also trying to downgrade from IOS XE 16, to the Cisco recommended release of 3.6.4 and ran into the exact same problem...

[christoffer]

I gave up and decided to run on 16.2.1. for now. Everything I tried based on the tips from this thread resulted in the same outcome :(

[Jorn Halsen Lukassen]

Hi.

Any progress in this case ? Did you contact TAC ?

I have run into the same issue, downgrading from Denali 16.1.2 to 3.6.5.