02-18-2021 06:45 PM
Hi guys, I have on hand a C3650-48FD-S that I just received from Cisco RMA, it comes with 16.3.8 firmware but the switch is meant to replace one from an existing stack that uses 3.6.8E. Every time I try to downgrade, it encounters boot failure and goes into a bootloop.
I've tried the following firmware: 3.6.8E and 3.6.7E, both install and bundle mode but none of those worked.
After each error I basically have to reload the 16.3.8 firmware through ROMMON.
I have found a similar thread that someone posted the bug code: CSCve02138
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve02138/?referring_site=bugquickviewredir
But I have not been able to access the information because it says I do not possess sufficient permission.
Here are the logs, it begins with me doing a md5 check on the firmware:
Switch#verify /md5 flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ............................................................................................................................................... ...................................................................................Done!
verify /md5 (flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin) = a04a54d69cb2b4d2867ed369e73598ae
Switch#$l file flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1
Expanding image file: flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
[1]: Expanding file
[1]: Finished expanding all-in-one software package in switch 1
SUCCESS: Finished expanding all-in-one software package.
[1]: Performing install
SUCCESS: install Finished
[1]: install package(s) on switch 1
--- Starting list of software package changes ---
Old files list:
Removed cat3k_caa-guestshell.16.03.08.SPA.pkg
Removed cat3k_caa-rpbase.16.03.08.SPA.pkg
Removed cat3k_caa-rpcore.16.03.08.SPA.pkg
Removed cat3k_caa-srdriver.16.03.08.SPA.pkg
Removed cat3k_caa-wcm.16.03.08.SPA.pkg
Removed cat3k_caa-webui.16.03.08.SPA.pkg
New files list:
Added cat3k_caa-base.SPA.03.06.08E.pkg
Added cat3k_caa-drivers.SPA.03.06.08E.pkg
Added cat3k_caa-infra.SPA.03.06.08E.pkg
Added cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
Added cat3k_caa-platform.SPA.03.06.08E.pkg
Added cat3k_caa-wcm.SPA.10.2.180.0.pkg
Finished list of software package changes
SUCCESS: Software provisioned. New software will load on reboot.
[1]: Finished install successful on switch 1
Checking status of install on [1]
[1]: Finished install in switch 1
SUCCESS: Finished install: Success on [1]
Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;
Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode= DEVICE
iPXE Timeout= 0
Switch#wr
Building configuration...
[OK]
Switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
*Feb 19 00:50:36.138: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.
Chassis 1 reloading, reason - Reload command
Feb 19 00:50:47.615 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Feb 19 00:50:58.239 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: process exit with reload stack code
octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system
Booting...
Interface GE 0 link down***ERROR: PHY link is down
Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 83298720
Warning: Signature envelope size exceed max. (512) allowed - 648, currently truncated to max.
Bundle Image
--------------------------------------
Kernel Address : 0x53777450
Kernel Size : 0x41838c/4293516
Initramfs Address : 0x53b8f7dc
Initramfs Size : 0xe781c2/15172034
Compression Format: .mzip
Bootable image at @ ram:0x53777450
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 623
Loading Linux kernel with entry point 0x8166a5e0 ...
Bootloader: Done loading app on core_mask: 0xf
### Launching Linux Kernel (flags = 0x5)
All packages are Digitally Signed
Starting System Services
FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
MCU type = 2 and image name is main_mcu_ucode_bundle_8_1_0.tar
Final tar file: main_mcu_ucode_bundle_8_1_0.tar
Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1
Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...84520 bytes.
Front-end Microcode IMG MGR: Programming device 0...wRrsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss!
Front-end Microcode IMG MGR: Microcode programming complete for device 0.
Front-end Microcode IMG MGR: Microcode programming complete in 42 seconds
Feb 19 00:55:13 %FED-3-INIT_FAILED: Module Pre-XCVR failed initiFeb 19 00:55:13 %IOSXE-2-PLATFORM: process ng_dumper: Process fed: terminated abnormally.
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: ===================== Exception Data ====================
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Exception in 'fed' with Signal=SIGABRT(6)
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Frame Pointer=0x7F96AD20, Program Counter=0x2F00E3B0
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed: Thread ID=5940: -Traceback= 1#09fe5074397bfe386e6907604b26df22 pthread:2EFFC000+123B0
Feb 19 00:55:14 %IOSXE-2-PLATFORM: process fed:
<Fri Feb 19 00:55:13 2021> Message from sysmgr: Reason Code:[2] Reset Reason:Service [fed] pid:[5940] terminated abnormally [6].
Service: fed
Description: Forwarding Engine Driver
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.06.08E.pkg//usr/binos/bin/fed
Started at Fri Feb 19 00:53:49 2021 (412898 us)
Stopped at Fri Feb 19 00:55:13 2021 (158324 us)
Uptime: 1 minutes 24 seconds
Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago
PID: 5940
Exit code: signal 6
CWD: /var/sysmgr/work
PID: 5940
UUID: 3005
get_local: failed: No such file or directory
Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.
02-18-2021 08:16 PM
Ok I have just tried to downgrade to 3.7.5E and it worked.
When I compared the two, I notice this line wasn't present when I boot with 3.7.5E
--------------------------------------------------------------------------------
Warning: Signature envelope size exceed max. (512) allowed - 648, currently truncated to max.
--------------------------------------------------------------------------------
Upon googling this line of error, I came across this bug code:
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg28749
There seems to be no solution given. Trying to downgrade again from 3.7.5E to 3.6.X gives the same error once again. I'm really at my wits end, do I need to RMA this unit again?
02-18-2021 11:14 PM
Have a look at CSCva22409.
Force the switch to boot into ROMMON and enter the following command:
flash_init boot flash:cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
It still does not work, try booting 3.7.X.
02-18-2021 11:22 PM - edited 02-18-2021 11:55 PM
Hi Leo, I have tried booting directly into 3.6.8E bundle mode through ROMMON but the result is still the same.
As my second post stated, I have successfully downgraded to 3.7.5E, but even from 3.7.5E i'm unable to further downgrade to 3.6.X without encountering boot failure.
Edit: Further attempts to downgrade to firmware 3.3.5SE has resulted in the same boot failure too.
02-19-2021 12:23 AM
@JasonChan27 wrote:
As my second post stated, I have successfully downgraded to 3.7.5E, but even from 3.7.5E i'm unable to further downgrade to 3.6.X without encountering boot failure.
This means you are indeed hitting CSCva22409.
02-19-2021 12:26 AM
Hi Leo, seeing as there is no workaround for the bug, does that mean I have no other choice other than upgrading the entire stack instead?
03-02-2021 02:29 PM - edited 03-02-2021 02:53 PM
Hi Jason,
Did you happen to find a solution for this? I am experiencing similar issue with exact same set of log messages while downgrading. I have a WS-C3650-24PD-L switch.
After reload it goes into a boot loop. However in my case I am downgrading from Cisco Everest 16.6.7 to 3.6.8E.
Not sure if CSCva22409 is applicable in my case as the factory loaded version is Everest 16.6.7
03-02-2021 04:30 PM
@ kronos wrote:
Not sure if CSCva22409 is applicable in my case as the factory loaded version is Everest 16.6.7
And why not? Bug ID specifically states switches will not be able to downgrade to 3.6.X (and earlier). The "lowest" level to downgrade is 3.7.X.
And you've stated you are downgrading to 3.6.8.
What happens if, say, you downgrade to 3.7.X?
03-02-2021 05:56 PM
Nope, there is no solution. TAC's advice was to upgrade my stack..
03-02-2021 08:00 PM - edited 03-02-2021 08:14 PM
@JasonChan27 wrote:
Nope, there is no solution. TAC's advice was to upgrade my stack..
Up to you.
I would be finding out if it is even possible to downgrade to 3.7.X.
If you have been unable to downgrade to 3.6.X but can downgrade to 3.7.X then this confirms you are hitting this bug.
03-02-2021 08:24 PM
Yeah, there's no solution even if I know I'm indeed hitting the bug. My whole stack was using 3.6.8E and just because of this bug(with no workaround or fix coming), I had to upgrade the entire stack. It's not like I can downgrade to 3.7.X and expect it to run with the rest of the machines still on 3.6.8
03-02-2021 08:56 PM
16.X.X on a 3650/3850 is not as "cracked up" as Cisco wants it to be.
Upgrading to 16.X.X is going to keep you up all night long.
This is why I'm going out on a limb and ask if you can go to 3.7.X.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide