Unable to downgrade C3650 to version 3.X

JasonChan27 · ‎02-18-2021

Hi guys, I have on hand a C3650-48FD-S that I just received from Cisco RMA, it comes with 16.3.8 firmware but the switch is meant to replace one from an existing stack that uses 3.6.8E. Every time I try to downgrade, it encounters boot failure and goes into a bootloop.

I've tried the following firmware: 3.6.8E and 3.6.7E, both install and bundle mode but none of those worked.

After each error I basically have to reload the 16.3.8 firmware through ROMMON.

I have found a similar thread that someone posted the bug code: CSCve02138

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve02138/?referring_site=bugquickviewredir

But I have not been able to access the information because it says I do not possess sufficient permission.

Here are the logs, it begins with me doing a md5 check on the firmware:

Switch#verify /md5 flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ...................................................................................Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin) = a04a54d69cb2b4d2867ed369e73598ae

Switch#$l file flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
[1]: Expanding file
[1]: Finished expanding all-in-one software package in switch 1
SUCCESS: Finished expanding all-in-one software package.
[1]: Performing install
SUCCESS: install Finished
[1]: install package(s) on switch 1
--- Starting list of software package changes ---
Old files list:
Removed cat3k_caa-guestshell.16.03.08.SPA.pkg
Removed cat3k_caa-rpbase.16.03.08.SPA.pkg
Removed cat3k_caa-rpcore.16.03.08.SPA.pkg
Removed cat3k_caa-srdriver.16.03.08.SPA.pkg
Removed cat3k_caa-wcm.16.03.08.SPA.pkg
Removed cat3k_caa-webui.16.03.08.SPA.pkg
New files list:
Added cat3k_caa-base.SPA.03.06.08E.pkg
Added cat3k_caa-drivers.SPA.03.06.08E.pkg
Added cat3k_caa-infra.SPA.03.06.08E.pkg
Added cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
Added cat3k_caa-platform.SPA.03.06.08E.pkg
Added cat3k_caa-wcm.SPA.10.2.180.0.pkg
Finished list of software package changes
SUCCESS: Software provisioned. New software will load on reboot.
[1]: Finished install successful on switch 1
Checking status of install on [1]
[1]: Finished install in switch 1
SUCCESS: Finished install: Success on [1]
Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode= DEVICE
iPXE Timeout= 0
Switch#wr
Building configuration...
[OK]
Switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]

*Feb 19 00:50:36.138: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
Chassis 1 reloading, reason - Reload command
Feb 19 00:50:47.615 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Feb 19 00:50:58.239 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: process exit with reload stack code

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

Booting...
Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 83298720
Warning: Signature envelope size exceed max. (512) allowed - 648, currently truncated to max.
Bundle Image
--------------------------------------
Kernel Address : 0x53777450
Kernel Size : 0x41838c/4293516
Initramfs Address : 0x53b8f7dc
Initramfs Size : 0xe781c2/15172034
Compression Format: .mzip

Bootable image at @ ram:0x53777450
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 623
Loading Linux kernel with entry point 0x8166a5e0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services

FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
MCU type = 2 and image name is main_mcu_ucode_bundle_8_1_0.tar
Final tar file: main_mcu_ucode_bundle_8_1_0.tar

Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...84520 bytes.
Front-end Microcode IMG MGR: Programming device 0...wRrsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss!
Front-end Microcode IMG MGR: Microcode programming complete for device 0.
Front-end Microcode IMG MGR: Microcode programming complete in 42 seconds
Feb 19 00:55:13 %FED-3-INIT_FAILED: Module Pre-XCVR failed initiFeb 19 00:55:13 %IOSXE-2-PLATFORM: process ng_dumper: Process fed: terminated abnormally.
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: ===================== Exception Data ====================
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Exception in 'fed' with Signal=SIGABRT(6)
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Frame Pointer=0x7F96AD20, Program Counter=0x2F00E3B0
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Thread ID=5940: -Traceback= 1#09fe5074397bfe386e6907604b26df22 pthread:2EFFC000+123B0
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed:

<Fri Feb 19 00:55:13 2021> Message from sysmgr: Reason Code:[2] Reset Reason:Service [fed] pid:[5940] terminated abnormally [6].
Service: fed
Description: Forwarding Engine Driver
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.06.08E.pkg//usr/binos/bin/fed

Started at Fri Feb 19 00:53:49 2021 (412898 us)
Stopped at Fri Feb 19 00:55:13 2021 (158324 us)
Uptime: 1 minutes 24 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 5940
Exit code: signal 6

CWD: /var/sysmgr/work

PID: 5940
UUID: 3005
get_local: failed: No such file or directory
Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.

JasonChan27 · ‎02-18-2021

Ok I have just tried to downgrade to 3.7.5E and it worked.

When I compared the two, I notice this line wasn't present when I boot with 3.7.5E

--------------------------------------------------------------------------------

Warning: Signature envelope size exceed max. (512) allowed - 648, currently truncated to max.

--------------------------------------------------------------------------------

Upon googling this line of error, I came across this bug code:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg28749

There seems to be no solution given. Trying to downgrade again from 3.7.5E to 3.6.X gives the same error once again. I'm really at my wits end, do I need to RMA this unit again?

Leo Laohoo · ‎02-18-2021

Have a look at CSCva22409.

Force the switch to boot into ROMMON and enter the following command:

flash_init
boot flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin

It still does not work, try booting 3.7.X.

JasonChan27 · ‎02-18-2021

Hi Leo, I have tried booting directly into 3.6.8E bundle mode through ROMMON but the result is still the same.

As my second post stated, I have successfully downgraded to 3.7.5E, but even from 3.7.5E i'm unable to further downgrade to 3.6.X without encountering boot failure.

Edit: Further attempts to downgrade to firmware 3.3.5SE has resulted in the same boot failure too.

Leo Laohoo · ‎02-19-2021

@JasonChan27 wrote:

As my second post stated, I have successfully downgraded to 3.7.5E, but even from 3.7.5E i'm unable to further downgrade to 3.6.X without encountering boot failure.

This means you are indeed hitting CSCva22409.

JasonChan27 · ‎02-19-2021

Hi Leo, seeing as there is no workaround for the bug, does that mean I have no other choice other than upgrading the entire stack instead?

kronos · ‎03-02-2021

Hi Jason,

Did you happen to find a solution for this? I am experiencing similar issue with exact same set of log messages while downgrading. I have a WS-C3650-24PD-L switch.

After reload it goes into a boot loop. However in my case I am downgrading from Cisco Everest 16.6.7 to 3.6.8E.

Not sure if CSCva22409 is applicable in my case as the factory loaded version is Everest 16.6.7

Leo Laohoo · ‎03-02-2021

@ kronos wrote:

Not sure if CSCva22409 is applicable in my case as the factory loaded version is Everest 16.6.7

And why not? Bug ID specifically states switches will not be able to downgrade to 3.6.X (and earlier). The "lowest" level to downgrade is 3.7.X.

And you've stated you are downgrading to 3.6.8.

What happens if, say, you downgrade to 3.7.X?

JasonChan27 · ‎03-02-2021

Nope, there is no solution. TAC's advice was to upgrade my stack..

Leo Laohoo · ‎03-02-2021

@JasonChan27 wrote:

Nope, there is no solution. TAC's advice was to upgrade my stack..

Up to you.

I would be finding out if it is even possible to downgrade to 3.7.X.
If you have been unable to downgrade to 3.6.X but can downgrade to 3.7.X then this confirms you are hitting this bug.

JasonChan27 · ‎03-02-2021

Yeah, there's no solution even if I know I'm indeed hitting the bug. My whole stack was using 3.6.8E and just because of this bug(with no workaround or fix coming), I had to upgrade the entire stack. It's not like I can downgrade to 3.7.X and expect it to run with the rest of the machines still on 3.6.8

Leo Laohoo · ‎03-02-2021

16.X.X on a 3650/3850 is not as "cracked up" as Cisco wants it to be.

Upgrading to 16.X.X is going to keep you up all night long.

This is why I'm going out on a limb and ask if you can go to 3.7.X.