Cisco 3650 no IOS - BRICK - upgrade failure

[michalfira]

Hi everyone

I'm trying to upload IOS into my Cisco 3650-48FS-S switch. Looks like it is bricked now :-(

I tried to boot from USB but I'm getting this kind of message:

switch: dir usbflash0:
Directory of usbflash0:/

4 -rw- 537489817 cat3k_caa-universalk9.16.03.07.SPA.bin

596320256 bytes available (410869760 bytes used)

switch:

switch:

switch: boot usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin
Reading full image into memory....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address : 0x5344d6a0
Kernel Size : 0x34e9ec/3467756
Initramfs Address : 0x5379c08c
Initramfs Size : 0x119e33d/18473789
Compression Format: mzip

Bootable image at @ ram:0x5344d6a0
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin" uncompressed and installed, entry point: 0x816903a0
Loading Linux kernel with entry point 0x816903a0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

>>> Boot Failed - pulling status and logs:
? mount_packages.service - Mount packages
 Loaded: loaded (/lib/systemd/system/mount_packages.service; static)
 Active: failed (Result: exit-code) since Sat 2018-12-29 18:41:04 Universal; 116ms ago
 Process: 435 ExecStart=/mount_packages.sh (code=exited, status=1/FAILURE)
 Main PID: 435 (code=exited, status=1/FAILURE)

[ 4.931993] localhost mount_packages.sh[435]: booted_item=/usbflash0/cat3k_caa-universalk9.16.03.07.SPA.bin
[ 6.221815] localhost mount_packages.sh[435]: Releasing iso memory
[ 6.468301] localhost mount_packages.sh[435]: /usr/bin/unbundle
[ 6.472839] localhost mount_packages.sh[435]: nova super packaging
[ 6.476495] localhost mount_packages.sh[435]: rp_super
[ 10.527577] localhost mount_packages.sh[435]: umount: /mnt/piso: not mounted
[ 11.243389] localhost mount_packages.sh[435]: Packages directory: [/tmp/sw/isos]
[ 11.244420] localhost mount_packages.sh[435]: Provisioning file: [/tmp/packages.conf]
[ 11.245414] localhost mount_packages.sh[435]: Provisioning file content:
[ 11.246380] localhost mount_packages.sh[435]: --/tmp/packages.conf------------------------
[ 11.248523] localhost mount_packages.sh[435]: sha1sum: 2aece2a25757fa84973dcd0699753673eade0ddd
[ 11.249503] localhost mount_packages.sh[435]: iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.07.SPA.pkg
[ 11.250488] localhost mount_packages.sh[435]: iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.251464] localhost mount_packages.sh[435]: iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.252480] localhost mount_packages.sh[435]: iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.253459] localhost mount_packages.sh[435]: iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.07.SPA.pkg
[ 11.254434] localhost mount_packages.sh[435]: iso rp 0 0 rp_webui cat3k_caa-webui.16.03.07.SPA.pkg
[ 11.255388] localhost mount_packages.sh[435]: iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.07.SPA.pkg
[ 11.256347] localhost mount_packages.sh[435]: iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.257336] localhost mount_packages.sh[435]: iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.07.SPA.pkg
[ 11.258325] localhost mount_packages.sh[435]: iso fp 0 0 fp cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.259354] localhost mount_packages.sh[435]: ----------------------==/tmp/packages.conf--
[ 11.472861] localhost mount_packages.sh[435]: Processing nfs directives ...
[ 11.493633] localhost mount_packages.sh[435]: Processing mount directives ...
[ 11.514300] localhost mount_packages.sh[435]: Processing iso directives ...
[ 11.595320] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_base/mount
[ 11.596331] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpbase.16.03.07.SPA.pkg
[ 11.813133] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_core/mount
[ 11.814252] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.869959] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_daemons/mount
[ 11.870949] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.916936] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_iosd/mount
[ 11.917931] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.141751] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_wcm/mount
[ 12.142773] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-wcm.16.03.07.SPA.pkg
[ 12.369260] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_webui/mount
[ 12.370278] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-webui.16.03.07.SPA.pkg
[ 12.458721] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/srdriver/mount
[ 12.459748] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-srdriver.16.03.07.SPA.pkg
[ 12.505824] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_security/mount
[ 12.506878] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.722834] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/guestshell/mount
[ 12.723864] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-guestshell.16.03.07.SPA.pkg
[ 12.769915] localhost mount_packages.sh[435]: ISO linking /tmp/sw/fp/0/0/fp/mount
[ 12.770947] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.783917] localhost mount_packages.sh[435]: Processing patch directives ...
[ 12.808444] localhost mount_packages.sh[435]: Waiting all backgroud processes
[ 12.809450] localhost mount_packages.sh[435]: Finished waiting backgroud processes
[ 17.904853] localhost mount_packages.sh[435]: no initramfs update required
[ 18.773468] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-srdriver.16.03.07.SPA.pkg is Digitally Signed
[ 19.055520] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-webui.16.03.07.SPA.pkg is Digitally Signed
[ 19.153564] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-guestshell.16.03.07.SPA.pkg is Digitally Signed
[ 19.234357] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-rpbase.16.03.07.SPA.pkg is Digitally Signed
[ 26.587076] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-wcm.16.03.07.SPA.pkg is Digitally Signed
[ 28.517723] localhost mount_packages.sh[435]: File /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg is corrupt.
[ 28.520804] localhost systemd[1]: mount_packages.service: main process exited, code=exited, status=1/FAILURE
[ 28.521835] localhost systemd[1]: Failed to start Mount packages.
[ 28.583238] localhost systemd[1]: Unit mount_packages.service entered failed state.
>>> Rebooting
octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

 

Booting...
Interface GE 0 link down***ERROR: PHY link is down

The system is about to execute the autoboot command.

Interrupt the system within 5 seconds to intervene.

flash:cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin: no such file or directory
Cannot open file flash:cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.


switch:

Any clue what was wrong?

My flash: folder looks like this:

switch: dir flash:
Directory of flash:/

7761 drwx 4096 .
2 drwx 4096 ..
7762 -rw- 2097152 nvram_config
7763 -rw- 2097152 nvram_config_bkup
7764 drwx 4096 .installer
7765 -rw- 37 bootloader_evt_handle.log
7766 drwx 4096 core
7767 drwx 4096 .prst_sync

1617453056 bytes available (32632832 bytes used)

switch:

Please help!!!

[Georg Pauwen]

Hello,

I think the syntax is:

switch: boot system usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin

[michalfira]

Negative - this command doesn't work
I put "system" after filename but got the same error during image install

[Georg Pauwen]

Hello,

just to be sure, did you format the USB from the router (format usbflash0:) ?

Also, what do you mean by:

I put "system" after filename but got the same error during image install

Which syntax did you use ?

[balaji.bandi]

There is 2 ways you can resolve the issue :

1. change the variable to boot from USB :

config t

!

boot system usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin

!

end

!

wr

2. 

copy usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin  flash:cat3k_caa-universalk9.16.03.07.SPA.bin

config t

!

boot system flash:/cat3k_caa-universalk9.16.03.07.SPA.bin

!

end

!

wr

reload and test it.

[michalfira]

Problem is switch doesn't boot so I can't do "enable", "config t" and other commands.
Each time it goes to "password recovery" mode and commands are very limited, so no config t is available.

Thanks

[balaji.bandi]

USB format should be FAT32
Copy your ios to the USB
connect the USB to switch
switch: boot usbflash0:/cat3k_caa-universalk9.16.03.07.SPA.bin

[michalfira]

go to my first post and read it. I just did that and it didn't work - that's why I'm asking if the switch is bricked and total garbage now?
It does load bin file into memory but after it's done and switch reboot itself it is trying to unpack files but there are some errors and failures.
And at the end it goes back to "password recovery mode" with "switch: " prompt

[michalfira · Accepted solution]

OK - I used different command and different IOS image file (older image)

switch: emergency-install usbflash0:<file.bin>

After that I got initial setup screen.

Image I used is: cat3k_caa-universalk9.SPA.03.06.09.E.152-2.E9.bin

Thanks

[balaji.bandi]

So this measn you are trying to  upgrading from cat3k_caa-universalk9.SPA.03.06.09.E.152-2.E9.bin to  to cat3k_caa-universalk9.16.03.07.SPA.bin ? is this correct ?

have you read the release notes for the upgrade ?

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-3/release_notes/ol-16-3-3850.html#pgfId-1135554

[meisemann7]

Is there an option for emergency-install or some kind of emergency install procedure 'not' using a USB/flash drive?  

[Leo Laohoo]

Yes, there is. 

Kindly take the time to read Xmodem Console Download Procedure Using ROMmon.

[balaji.bandi]

I have read your post, based on that we are here to help you based on the information provided and  possible options to resolve. (if you think this emergency you need to Open TAC case i belive). 

coming back to basic question for the below :

"I'm trying to upload IOS into my Cisco 3650-48FS-S switch. Looks like it is bricked now."

is this means you try to upgrade ? or failed device you try to recover. ?

if the failed device you try to recover, i beliee you need to try reset the device  to factory reset.

http://wannabelab.blogspot.com/2017/01/password-recovery-on-cisco-3650.html

[Leo Laohoo]

Question:  Was this switch loaded with 3.X.X before?  

If yes, then you'll need to do Emergency Install prcoceedure. 

It is stated in the 16.3.X Release Notes that:

---

Starting from 16.3.5 release, you will not be able to boot Cisco IOS XE Denali 16.3.5 in bundle mode via flash drive for the first time with a Cisco IOS XE 3.xE boot loader. The Cisco IOS XE 3.xE boot loaders have a limitation that they cannot boot an image larger than 512MB via flash.

---

[NetworkDude255]

Hi Michalfira, in this scenario, I'd recommend loading the IOS you have on the USB to a local TFTP server and booting that IOS from the rommon prompt.

Here is a link below that I've followed with success previously.

https://community.cisco.com/t5/networking-blogs/booting-cisco-router-from-a-tftp-server/ba-p/3101600

Good Luck!