UPGRADE WS-C3650-48TQ-S

[fernando.cruz]

I need to update a WS-C3650-48TQ-S, which has the iOS version 3.3.4SE.

Currently the recommended version is: Fuji-16.9.4 MD.

Can I make the jump from version 3.3.4 to version 16.9.4
Or in your case what would be the best option

[acampbell]

Hi,

Please look at this config guide:-

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/16-9/configuration_guide/sys_mgmt/b_169_sys_mgmt_3650_cg/upgrading_the_switch_software.html

[balaji.bandi]

here is the upgrade procedure :

make sure you read the release notes and pre-requisites before upgrade to Fuji.

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3650/software/release/16-9/release_notes/ol-16-9-3650.html#id_67367

[Leo Laohoo · Accepted solution]

Read the Release Notes carefully.
You can upgrade direct from 3.X to 16.X.X, however, there will be a mandatory micro-code upgrade so the initial reboot may take up to 22 minutes.

[fernando.cruz]

Hi Leo.

It would be best to make an intermediate jump between the installed version and the recommended version (question)
If so, how do I know which is the appropriate intermediate version (question)

[Leo Laohoo]

I never said anything about "intermediate version".
I would not recommend going to an "intermediate" version when it is not necessary.

[Keith_Wood]

Not to hijack the thread, but I just ran the upgrade process on a 3850 going straight from 3.x to 16.x and it now fails to boot. Here's the full process:

Switch#$ tftp://10.255.31.108/cat3k_caa-universalk9.16.09.04.SPA.bin
Preparing install operation ...
[1]: Downloading file tftp://10.255.31.108/cat3k_caa-universalk9.16.09.04.SPA.bin to active switch 1
[1]: Finished downloading file tftp://10.255.31.108/cat3k_caa-universalk9.16.09.04.SPA.bin to active switch 1
[1]: Starting install operation
[1]: Expanding bundle cat3k_caa-universalk9.16.09.04.SPA.bin
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding bundle cat3k_caa-universalk9.16.09.04.SPA.bin
[1]: Verifying and copying expanded package files to flash:
[1]: Verified and copied expanded package files to flash:
[1]: Starting compatibility checks
[1]: Finished compatibility checks
[1]: Starting application pre-installation processing
[1]: Finished application pre-installation processing
[1]: Old files list:
Removed cat3k_caa-base.SPA.03.02.01.SE.pkg
[1]: New files list:
Added cat3k_caa-guestshell.16.09.04.SPA.pkg
Added cat3k_caa-rpbase.16.09.04.SPA.pkg
Added cat3k_caa-rpcore.16.09.04.SPA.pkg
Added cat3k_caa-srdriver.16.09.04.SPA.pkg
Added cat3k_caa-webui.16.09.04.SPA.pkg
[1]: Creating pending provisioning file
[1]: Finished installing software. New software will load on reboot.
[1]: Committing provisioning file

[1]: Do you want to proceed with reload? [yes/no]: yes

System configuration has been modified. Save? [yes/no]: no
[1]: Reloading

Switch#
*Nov 2 07:58:34.696: %STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr: Received reload request for switch 1, reason User requested reload
*Nov 2 07:58:34.696: %STACKMGR-1-RELOAD: 1 stack-mgr: Reloading due to reason User requested reload
*Nov 2 07:58:35.246: %IOSXE-3-PLATFORM: 1 process sysmgr: Reset/Reload requested by [stack-manager].
<Sat Nov 2 07:58:35 2019> Message from sysmgr: Reset Reason:Reset/Reload requested by [stack-manager]. [User requested reload]
Unmounting ng3k filesystems...
Unmounted /dev/sda3...
Warning! - some ng3k filesystems may not have unmounted cleanly...
Please stand by while rebooting the system...
Restarting system.

Booting...Initializing and Testing RAM ++++@@@@####...################################++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@++@@done.
Memory Test Pass!

Base ethernet MAC Address: 20:bb:c0:00:92:80
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 2 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 30894576
Nova Bundle Image
--------------------------------------
Kernel Address : 0x6042e3b4
Kernel Size : 0x3684e0/3572960
Initramfs Address : 0x60796894
Initramfs Size : 0x19b595c/26958172
Compression Format: mzip

Bootable image at @ ram:0x6042e3b4
Bootable image segment 0 address range [0x81100000, 0x81c14630] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 380
Loading Linux kernel with entry point 0x816e4a80 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

%IOSXEBOOT-5c8e9d6656e9d89a8dedeae457871084-new_cksum: (rp/0): 4
%IOSXEBOOT-5c8e9d6656e9d89a8dedeae457871084-saved_cksum: (rp/0): 4
%IOSXEBOOT-Sat-###: (rp/0): Nov 2 08:03:29 Universal 2019 PLEASE DO NOT POWER CYCLE ### BOOT LOADER UPGRADING 4
● tdl.service - TDL Resolve
Loaded: loaded (/lib/systemd/system/tdl.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-11-02 08:03:35 Universal; 70ms ago
Process: 3945 ExecStart=/usr/binos/conf/tdl_boottime.sh (code=exited, status=1/FAILURE)
Main PID: 3945 (code=exited, status=1/FAILURE)

[ 65.599528] localhost systemd[1]: Starting TDL Resolve...
[ 84.926712] RP_0 tdl_boottime.sh[3945]: luajit: /usr/binos/conf/epoch_lib.lua:798: attempt to compare two nil values
[ 84.927565] RP_0 tdl_boottime.sh[3945]: stack traceback:
[ 84.928367] RP_0 tdl_boottime.sh[3945]: /usr/binos/conf/epoch_lib.lua:798: in function 'add_and_merge_record'
[ 84.929117] RP_0 tdl_boottime.sh[3945]: /usr/binos/conf/epoch_lib.lua:816: in function 'generate_merged_metadata'
[ 84.929881] RP_0 tdl_boottime.sh[3945]: /usr/binos/conf/epoch_lib.lua:906: in function 'generate_domain'
[ 84.930656] RP_0 tdl_boottime.sh[3945]: /usr/binos/conf/epoch_resolve.lua:209: in main chunk
[ 84.931448] RP_0 tdl_boottime.sh[3945]: [C]: at 0xaab6e74cb0
[ 84.941553] RP_0 tdl_boottime.sh[3945]: Time to check parameters: 0.001 seconds.
[ 84.942353] RP_0 tdl_boottime.sh[3945]: Time to build package table: 0.002 seconds.
[ 84.943132] RP_0 tdl_boottime.sh[3945]: Time to cache objects: 7.634 seconds.
[ 84.950722] RP_0 systemd[1]: tdl.service: Main process exited, code=exited, status=1/FAILURE
[ 84.952662] RP_0 systemd[1]: Failed to start TDL Resolve.
[ 84.962066] RP_0 systemd[1]: tdl.service: Unit entered failed state.
[ 84.962846] RP_0 systemd[1]: tdl.service: Failed with result 'exit-code'.
Nov 2 08:03:36.012: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task tdl.service ( exit-code )
>>> Rebooting
Failed to set wall message, ignoring: The name org.freedesktop.login1 was not provided by any .service files
Failed to reboot system via logind: The name org.freedesktop.login1 was not provided by any .service files
octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system
▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒

(lots more lines of this)

▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒BAUD=9600
BOOT=flash:packages.conf
CFG_MODEL_NUM=WS-C3850-24P-S
CLEI_CODE_NUMBER=IPMVE00BRB
ECI_CODE_NUMBER=465444
LINUX_COREMASK=15
MAC_ADDR=20:BB:C0:00:92:80
MANUAL_BOOT=no
MODEL_NUM=WS-C3850-24P
MODEL_REVISION_NUM=A0
MOTHERBOARD_ASSEMBLY_NUM=73-14441-06
MOTHERBOARD_REVISION_NUM=A0
MOTHERBOARD_SERIAL_NUM=<redacted>
POE1_ASSEMBLY_NUM=73-14095-01
POE1_REVISION_NUM=A0
POE1_SERIAL_NUM=FOC1712582F
RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin
STKPWR_ASSEMBLY_NUM=73-11956-08
STKPWR_REVISION_NUM=A0
STKPWR_SERIAL_NUM=<redacted>
SYSTEM_SERIAL_NUM=<redacted>
TAN_NUM=800-37562-01
TAN_REVISION_NUMBER=B0
TERMLINES=0
USB_ASSEMBLY_NUM=73-12923-05
USB_REVISION_NUM=A0
USB_SERIAL_NUM=<redacted>
VERSION_ID=V02
TEMPLATE=advanced
BSI=0
SWITCH_NUMBER=1
RANDOM_NUM=763806977
ABNORMAL_RESET_COUNT=0

Now even holding Mode doesn't get back to the switch: prompt. It just hangs on loading.

[Leo Laohoo]

---

@Keith_Wood wrote:

[ 84.950722] RP_0 systemd[1]: tdl.service: Main process exited, code=exited, status=1/FAILURE
[ 84.952662] RP_0 systemd[1]: Failed to start TDL Resolve.
[ 84.962066] RP_0 systemd[1]: tdl.service: Unit entered failed state.
[ 84.962846] RP_0 systemd[1]: tdl.service: Failed with result 'exit-code'.
Nov 2 08:03:36.012: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task tdl.service ( exit-code )

---

Keith, 

Read this:  Emergency Recovery

[Keith_Wood]

Thanks for the reply! I have read that a few times. The last thing I said in the post above is that holding mode doesn't bring up the switch: prompt. It just cycles through the last part of the failure messages and then hangs.

[Leo Laohoo]

Pull the power cable, wait for 5 seconds, plug the power cable back, leave it "hanging" for 20 minutes.
If the switch is still unresponsive, contact Cisco TAC and organize for an RMA.

[AntoineDefossez]

Hello,

I got the same issue last week.

The "new" and "force" key word behid the classical "software install file flash:X.bin" are really important.

I reallize an upgrade in LAB. Upgrading to 16.9.5 from 3.6.8E on two 3850 and it works :

Switch#$stall file flash:cat3k_caa-universalk9.16.09.05.SPA.bin new force
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1 2]: Expanding bundle flash:cat3k_caa-universalk9.16.09.05.SPA.bin
[1 2]: Copying package files
[1 2]: Package files copied
[1 2]: Finished expanding bundle flash:cat3k_caa-universalk9.16.09.05.SPA.bin
[1 2]: Verifying and copying expanded package files to flash:
[1 2]: Verified and copied expanded package files to flash:
[1 2]: Starting compatibility checks
[1 2]: Bypassing peer package compatibility checks due to 'force' command option
[1 2]: Finished compatibility checks
[1 2]: Starting application pre-installation processing
[1 2]: Finished application pre-installation processing
[1]: Old files list:
Removed cat3k_caa-base.SPA.03.06.08E.pkg
Removed cat3k_caa-drivers.SPA.03.06.08E.pkg
Removed cat3k_caa-infra.SPA.03.06.08E.pkg
Removed cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
Removed cat3k_caa-platform.SPA.03.06.08E.pkg
Removed cat3k_caa-wcm.SPA.10.2.180.0.pkg
[2]: Old files list:
Removed cat3k_caa-base.SPA.03.06.08E.pkg
Removed cat3k_caa-drivers.SPA.03.06.08E.pkg
Removed cat3k_caa-infra.SPA.03.06.08E.pkg
Removed cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
Removed cat3k_caa-platform.SPA.03.06.08E.pkg
Removed cat3k_caa-wcm.SPA.10.2.180.0.pkg
[1]: New files list:
Added cat3k_caa-guestshell.16.09.05.SPA.pkg
Added cat3k_caa-rpbase.16.09.05.SPA.pkg
Added cat3k_caa-rpcore.16.09.05.SPA.pkg
Added cat3k_caa-srdriver.16.09.05.SPA.pkg
Added cat3k_caa-webui.16.09.05.SPA.pkg
[2]: New files list:
Added cat3k_caa-guestshell.16.09.05.SPA.pkg
Added cat3k_caa-rpbase.16.09.05.SPA.pkg
Added cat3k_caa-rpcore.16.09.05.SPA.pkg
Added cat3k_caa-srdriver.16.09.05.SPA.pkg
Added cat3k_caa-webui.16.09.05.SPA.pkg
[1 2]: Creating pending provisioning file
[1 2]: Finished installing software. New software will load on reboot.
[1 2]: Committing provisioning file

[1 2]: Do you want to proceed with reload? [yes/no]: yes
[1 2]: Reloading

Microcode takes 12-13 mins to upgrade.