Cisco 3850: IOS-XE/Firmware Upgrade

[Leo Laohoo]

Cisco 3850:  IOS-XE/Firmware Upgrade

(Install Mode)

NOTE:

• This procedure is aimed at Cisco 3850 switch ONLY.
• IOS-XE Bundle Mode is not covered. 
• 9300, 9500 (vanilla & high-performance), ISR 1k, ISR 4k and ASR is not covered. 
• Router(s) & switch(es) running “classic” IOS is not covered.

A Word from our Sponsors:

There is a very fine line between writing something short-and-sweet and writing something comprehensive while losing the target “market” in the process.  I have done all I can to make this document as “digestible” as possible without losing much of the important content but still maintaining some forward momentum. 

Introduction: 

Switch and/or router firmware had (past tense) a simple procedure, however, for IOS-XE each platform has different commands, process & procedures, and “gotchas”.  Because of this, I have broken up the process into different sections:  9300, 9500 & routers.

Section 1:  General Procedure

Section 2:  Install SMU

Section 3:  GOTCHA

Section 4:  Emergency-Install

Section 1:  General Procedure

1. Read the Release Notes very carefully.
2. IOS-XE firmware and SMU files have a filename extension of “bin”. 
3. Clean the flash:  install remove inactive

NOTE:  Switch/stack on IOS-XE version 3.X.X, uses a different command:

software clean force

4. Copy the file into the switch/stack.
5. Make sure the boot-variable string is pointing to “packages.conf” file. 

6. Initiate Install Mode (Non-Disruptive/Do NOT Reboot 

request platform software package install switch all file flash:filename.bin on-reboot new auto-copy verbose

NOTE:  If upgrading from 3.X.X, the command is different: 

software install file flash: filename.bin new force verb on-reboot

IMPORTANT: 

• If the file extraction is successful (or not), the last two lines will display the result. 
• Go to the GOTCHA section if the following (image below) appears:

7. Inspect the contents “packages.conf” file.  Is this the intended version or not?  If NO, go to the GOTCHA section.  Command:  more flash:packages.conf | begin for NOVA

IMPORTANT: 

• If the switches are in a stack, inspect the “packages.conf” of each switch stack member.

more flash-1:packages.conf | begin for NOVA

more flash-2:packages.conf | begin for NOVA

more flash-3:packages.conf | begin for NOVA

...

more flash-8:packages.conf | begin for NOVA

• If the switch is standalone, the command is: 

more flash:packages.conf | begin for NOVA

8. REBOOT the switch/stack.
9. [RECOMMENDED][OPTIONAL] Clean the flash:  install remove inactive
10. DONE.

Section 2:  Install SMU

Software Maintenance Update (aka “patch”)

WARNING:

• Only routers and switches running IOS-XE support SMU. 
• Router(s) & switch(es) running “classic” IOS do not support SMU.
• IMPORTANT:  Always treat SMU update as disruptive (reboots the switch/stack). 
• IMPORTANT:  SMU file is platform-specific.
• IMPORTANT:  SMU file is version-specific. 

Image (above):  Sample SMU filename

Image (above):  Attempting to install an SMU meant for a different version.

Proceedure

1. Read the Release Notes very carefully.
2. SMU firmware have a filename extension of “bin”. 
3. Copy the SMU file into the switch. 
4. Install the SMU: 

install add file flash:SMU_filename.bin activate commit

WARNING:  Immediately after entering “y” may/will cause the router &/or switch to reboot.

5. Verify the SMU has been successfully installed.

• Fast Method:  sh version | begin Active SMU

• Slow Method:  sh install summary

6. DONE.

Section 3:  GOTCHA

1. If the message (image below) appears at the end of the package extraction process THIS IS NOT A FALSE POSITIVE. 

2. Check the contents of the flash/bootflash and compare the date stamp of the extracted packages against the date stamp of the “packages.conf” file (see image below). 

3. Look in the flash/bootflash directory for two (2) files with an extension of “conf”.  (Image below)

4. First, rename “packages.conf” to, say, “packages.conf.bak”.
5. Next, rename the firmware.conf to “packages.conf” (see below).

 

6. Check the contents of the new “packages.conf” file.  Is this the target version?   

 

7. Inspect the contents of each “packages.conf” file of every member of the stack. 
8. Make sure the boot-variable string is pointing to the “packages.conf” file.
9. DONE. 
   

Section 4:  Emergency-Install

WARNING:  Emergency-Install command will erase the flash/bootflash of the appliance. 

(Make sure the config is stored somewhere else.)

1. Emergency-Install is to be used when the switch boots into ROMMON. 
2. The “emergency-install” command can only be used in ROMMON. 
3. Emergency-Install process can only be effectively achieved by using a USB thumb drive.

Image above shows failed “emergency-install” because the firmware is in the flash. 

WARNING:  Emergency-Install command will erase the flash/bootflash of the appliance.  

4. Command:  emergency-install usbflash0:filename.bin
5. DONE.

[Ehsan Momeni Bashusqeh]

Great, Thanks for sharing.

[Heinz]

Hi Leo,

What SSH client are you using?

Regards

[Leo Laohoo]

SecureCRT

[Ehsan Momeni Bashusqeh]

thanks a lot for detailed explanation

[winfil2107804]

Hello

i have just upgraded 3850 Core switche stack, and wanted to report a remark, during upgrade process,

Instead of using the command:

*boot system switch all flash:cat3k_caa-universalk9.16.12.05b.SPA.bin

It is better to use the following commands, specifying the switch ID.

*(config)#boot system switch 1 flash:cat3k_caa-universalk9.16.12.05b.SPA.bin

*(config)#boot system switch 2 flash:cat3k_caa-universalk9.16.12.05b.SPA.bin

Because the first command strangely places the firmware in second boot !!! So it does not take the new version on the next reboot.

BR

Youssef

[Leo Laohoo]

@winfil2107804, 

1.  Why would I want to specify what each member of the switch what to boot when I can perform a single command from the switch master? 

2.  The command you mentioned is for Bundle Mode.  This entire process talks about Install Mode.  

[winfil2107804]

@Leo Laohoo 

yes, what i report concern Bundle Mode. i just wanted to share this with users could have the same issue.

BR

Youssef

[oogii-badral-7]

Booting...(use DDR clock 667 MHz)Initializing RAM +++++++@@@@@@@@++++++++++++++++++++++++++++++++Up 1000 Mbps Full duplex (port 0) (SGMII)

flashfs[7]: Checking block 43...bad block number (19551)
flashfs[7]: erasing block 43...done.
flashfs[7]: Checking block 44...bad block number (770)
flashfs[7]: erasing block 44...
flashfs[7]: erasing block, flash handle 0x534006f4, device 0, offset 0x3f0000: Operation Failed
The system encountered an error during initialization of the filesystem.
The following commands will initialize the filesystem, and finish
loading the operating system software:

flash_init
boot

switch: set
?=
ABNORMAL_RESET_COUNT=
ASIC_PCI_RESET=1
BOOT=usbflash0:/cat3k_caa-universalk9.16.12.10a.SPA.bin
BSI=0
CFG_MODEL_NUM=WS-C3850-24S-S
CLEI_CODE_NUMBER=IPMWZ00ARB
CONFIG_FILE=flash:
DDR_SPEED=667
DEFAULT_ROUTER=192.168.0.1
D_STACK_DOMAIN_NUM=1
ECI_CODE_NUMBER=468639
IP_ADDR=192.168.0.200/255.255.255.0
LICENSE_BOOT_LEVEL=ipbasek9,all:ngwc;
MAC_ADDR=70:01:B5:B5:08:00
MANUAL_BOOT=1
MODEL_NUM=WS-C3850-24S
MODEL_REVISION_NUM=L0
MOTHERBOARD_ASSEMBLY_NUM=73-15839-07
MOTHERBOARD_REVISION_NUM=B0
MOTHERBOARD_SERIAL_NUM=FOC22114ZQ7
RANDOM_NUM=1411370130
RECOVERY_BUNDLE=
RET_2_RCALTS=1127404891
SKIP_POST=no
STKPWR_ASSEMBLY_NUM=73-11956-08
STKPWR_REVISION_NUM=B0
STKPWR_SERIAL_NUM=FOC22107WLR
SWITCH_IGNORE_STARTUP_CFG=0
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=FCW2205D1CN
TAN_NUM=800-41089-06
TAN_REVISION_NUMBER=F0
TEMPLATE=advanced
TERMLINES=0
USB_ASSEMBLY_NUM=73-16167-02
USB_REVISION_NUM=A0
USB_SERIAL_NUM=FOC221145WD
VERSION_ID=V02

switch: boot
Reading full image into memory................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address : 0x537956f0
Kernel Size : 0x43857f/4425087
Initramfs Address : 0x53bcdc6f
Initramfs Size : 0x1ca9a56/30054998
Compression Format: mzip

Bootable image at @ ram:0x537956f0
Bootable image segment 0 address range [0x81100000, 0x81da6400] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:/cat3k_caa-universalk9.16.12.10a.SPA.bin" uncompressed and installed, entry point: 0x818968e0
Loading Linux kernel with entry point 0x818968e0 ...
Bootloader: Done loading app on core_mask: 0x3f

### Launching Linux Kernel (flags = 0x5)

Linux version 4.9.187 (deeratho@sjc-ads-7586) (gcc version 5.3.0 (GCC) ) #1 SMP Sun Jul 2 23:04:41 PDT 2023
CVMSEG size: 2 cache lines (256 bytes)
Cavium Inc. SDK-5.1.0
bootconsole [early0] enabled
CPU0 revision is: 000d900a (Cavium Octeon II)
Checking for the multiply/shift bug... no.
Checking for the daddiu bug... no.
%IOSXEBOOT-65259848eff01324a205b969dcebb486-new_cksum: (rp/0): 4
%IOSXEBOOT-65259848eff01324a205b969dcebb486-saved_cksum: (rp/0): 4
%IOSXEBOOT-Sun-###: (rp/0): Sep 25 01:59:30 Universal 2005 PLEASE DO NOT POWER CYCLE ### BOOT LOADER UPGRADING 4
Mainboard hardware authentication failed. Abort init ...

^CMainboard hardware authentication failed. Abort init ...

Mainboard hardware authentication failed. Abort init ...

[Leo Laohoo]

---

@oogii-badral-7  wrote:

Mainboard hardware authentication failed. Abort init ...
Mainboard hardware authentication failed. Abort init ...
Mainboard hardware authentication failed. Abort init ...​

---

RMA the switch.

[jason-m-laidlaw-ctr]

This post was helpful and what to say I just restored a c3850 48XS out of ROMMON with the tftp command: emergency-install tftp://XXX.XXX.XXX.XX/filename.bin

Make sure you can ping the tftp server ip from your Switch while plugged into the MGMT port, not console port. 

Step 1 From your PC, download the software image file (image.bin) from Cisco.com.
Step 2 Load the software image to your TFTP server.
Step 3 Connect your PC to the switch Ethernet management port.
Step 4 Unplug the switch power cord.
Step 5 Press the Mode button, and at the same time, reconnect the power cord to the switch.
Step 6 From the bootloader (ROMMON) prompt, ensure that you can ping your TFTP server.
a) Set the IP address switch: set IP_ADDR ip_address subnet_mask
Example:
switch: set IP_ADDR 192.0.2.123/255.255.255.0
b) Set the default router IP address switch: set DEFAULT_ROUTER ip_address
Example:
switch: set DEFAULT_ROUTER 192.0.2.1
c) Verify that you can ping the TFTP server switch: ping ip_address_of_TFTP_server
Example:
switch: ping 192.0.2.15
ping 192.0.2.1 with 32 bytes of data...
Host 192.0.2.1 is alive.
switch:
Step 7 Verify that you have a recovery image in your recovery partition (sda9:).
This recovery image is required for recovery using the emergency-install feature.
Example:
switch: dir sda9:
Directory of sda9:/
2 drwx 1024 .
2 drwx 1024 ..
11 -rw- 18923068 c3850-recovery.bin
36939776 bytes available (20830208 bytes used)
switch:
Step 8 From the bootloader (ROMMON) prompt, initiate the emergency-install feature that assists you in recovering the software
image on your switch.
WARNING: The emergency install command will erase your entire boot flash!
Example:
Switch#
emergency-install
tftp://192.0.2.47/cat3k_caa-universalk9.SSA.03.12.02.EZP.150-12.02.EZP.150-12.02.EZP.bin
The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery
(tftp://192.0.2.47/cat3k/cat3k_caa-universalk9.SPA.03.02.00.SE.150-1.EX.bin)...