Solved: Cisco 3650 no IOS - BRICK - upgrade failure - Page 2

michalfira · ‎12-29-2018

Hi everyone

I'm trying to upload IOS into my Cisco 3650-48FS-S switch. Looks like it is bricked now :-(

I tried to boot from USB but I'm getting this kind of message:

switch: dir usbflash0:
Directory of usbflash0:/

4 -rw- 537489817 cat3k_caa-universalk9.16.03.07.SPA.bin

596320256 bytes available (410869760 bytes used)

switch:

switch:

switch: boot usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin
Reading full image into memory....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address : 0x5344d6a0
Kernel Size : 0x34e9ec/3467756
Initramfs Address : 0x5379c08c
Initramfs Size : 0x119e33d/18473789
Compression Format: mzip

Bootable image at @ ram:0x5344d6a0
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.16.03.07.SPA.bin" uncompressed and installed, entry point: 0x816903a0
Loading Linux kernel with entry point 0x816903a0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

>>> Boot Failed - pulling status and logs:
? mount_packages.service - Mount packages
 Loaded: loaded (/lib/systemd/system/mount_packages.service; static)
 Active: failed (Result: exit-code) since Sat 2018-12-29 18:41:04 Universal; 116ms ago
 Process: 435 ExecStart=/mount_packages.sh (code=exited, status=1/FAILURE)
 Main PID: 435 (code=exited, status=1/FAILURE)

[ 4.931993] localhost mount_packages.sh[435]: booted_item=/usbflash0/cat3k_caa-universalk9.16.03.07.SPA.bin
[ 6.221815] localhost mount_packages.sh[435]: Releasing iso memory
[ 6.468301] localhost mount_packages.sh[435]: /usr/bin/unbundle
[ 6.472839] localhost mount_packages.sh[435]: nova super packaging
[ 6.476495] localhost mount_packages.sh[435]: rp_super
[ 10.527577] localhost mount_packages.sh[435]: umount: /mnt/piso: not mounted
[ 11.243389] localhost mount_packages.sh[435]: Packages directory: [/tmp/sw/isos]
[ 11.244420] localhost mount_packages.sh[435]: Provisioning file: [/tmp/packages.conf]
[ 11.245414] localhost mount_packages.sh[435]: Provisioning file content:
[ 11.246380] localhost mount_packages.sh[435]: --/tmp/packages.conf------------------------
[ 11.248523] localhost mount_packages.sh[435]: sha1sum: 2aece2a25757fa84973dcd0699753673eade0ddd
[ 11.249503] localhost mount_packages.sh[435]: iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.07.SPA.pkg
[ 11.250488] localhost mount_packages.sh[435]: iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.251464] localhost mount_packages.sh[435]: iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.252480] localhost mount_packages.sh[435]: iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.253459] localhost mount_packages.sh[435]: iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.07.SPA.pkg
[ 11.254434] localhost mount_packages.sh[435]: iso rp 0 0 rp_webui cat3k_caa-webui.16.03.07.SPA.pkg
[ 11.255388] localhost mount_packages.sh[435]: iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.07.SPA.pkg
[ 11.256347] localhost mount_packages.sh[435]: iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.257336] localhost mount_packages.sh[435]: iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.07.SPA.pkg
[ 11.258325] localhost mount_packages.sh[435]: iso fp 0 0 fp cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.259354] localhost mount_packages.sh[435]: ----------------------==/tmp/packages.conf--
[ 11.472861] localhost mount_packages.sh[435]: Processing nfs directives ...
[ 11.493633] localhost mount_packages.sh[435]: Processing mount directives ...
[ 11.514300] localhost mount_packages.sh[435]: Processing iso directives ...
[ 11.595320] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_base/mount
[ 11.596331] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpbase.16.03.07.SPA.pkg
[ 11.813133] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_core/mount
[ 11.814252] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.869959] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_daemons/mount
[ 11.870949] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 11.916936] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_iosd/mount
[ 11.917931] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.141751] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_wcm/mount
[ 12.142773] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-wcm.16.03.07.SPA.pkg
[ 12.369260] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_webui/mount
[ 12.370278] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-webui.16.03.07.SPA.pkg
[ 12.458721] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/srdriver/mount
[ 12.459748] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-srdriver.16.03.07.SPA.pkg
[ 12.505824] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/rp_security/mount
[ 12.506878] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.722834] localhost mount_packages.sh[435]: ISO linking /tmp/sw/rp/0/0/guestshell/mount
[ 12.723864] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-guestshell.16.03.07.SPA.pkg
[ 12.769915] localhost mount_packages.sh[435]: ISO linking /tmp/sw/fp/0/0/fp/mount
[ 12.770947] localhost mount_packages.sh[435]: -> /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg
[ 12.783917] localhost mount_packages.sh[435]: Processing patch directives ...
[ 12.808444] localhost mount_packages.sh[435]: Waiting all backgroud processes
[ 12.809450] localhost mount_packages.sh[435]: Finished waiting backgroud processes
[ 17.904853] localhost mount_packages.sh[435]: no initramfs update required
[ 18.773468] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-srdriver.16.03.07.SPA.pkg is Digitally Signed
[ 19.055520] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-webui.16.03.07.SPA.pkg is Digitally Signed
[ 19.153564] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-guestshell.16.03.07.SPA.pkg is Digitally Signed
[ 19.234357] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-rpbase.16.03.07.SPA.pkg is Digitally Signed
[ 26.587076] localhost mount_packages.sh[435]: Package /tmp/sw/isos/cat3k_caa-wcm.16.03.07.SPA.pkg is Digitally Signed
[ 28.517723] localhost mount_packages.sh[435]: File /tmp/sw/isos/cat3k_caa-rpcore.16.03.07.SPA.pkg is corrupt.
[ 28.520804] localhost systemd[1]: mount_packages.service: main process exited, code=exited, status=1/FAILURE
[ 28.521835] localhost systemd[1]: Failed to start Mount packages.
[ 28.583238] localhost systemd[1]: Unit mount_packages.service entered failed state.
>>> Rebooting
octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

 

Booting...
Interface GE 0 link down***ERROR: PHY link is down

The system is about to execute the autoboot command.

Interrupt the system within 5 seconds to intervene.

flash:cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin: no such file or directory
Cannot open file flash:cat3k_caa-universalk9.SPA.03.07.05.E.152-3.E5.bin

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.


switch:

Any clue what was wrong?

My flash: folder looks like this:

switch: dir flash:
Directory of flash:/

7761 drwx 4096 .
2 drwx 4096 ..
7762 -rw- 2097152 nvram_config
7763 -rw- 2097152 nvram_config_bkup
7764 drwx 4096 .installer
7765 -rw- 37 bootloader_evt_handle.log
7766 drwx 4096 core
7767 drwx 4096 .prst_sync

1617453056 bytes available (32632832 bytes used)

switch:

Please help!!!

michalfira · ‎12-29-2018

Ok - story is I had to do password recovery/default to factory settings the 3650 switch. I ran the Cisco procedure (press and hold mode button during startup). Unfortunately there was no packages.conf file so I decided to upload a new image into the switch. After the image was uploaded everything was fine but it was in bundle mode and I wanted to have it in install mode. When tried to upload different image something went wrong (don’t ask me what went wrong - it was at 3AM) and I couldn’t upload any image.

I totally forgot about emergency-install command and the lowest possible IOS image that was available to download from Cisco website.

Now all is good. Switch works good with clean config. That’s what I wanted.

Thanks for all your pieces of advice.

Happy New Year.

P.S.

It was a switch, not a router so router recovery tips didn’t work :-)

balaji.bandi · ‎12-30-2018

Glad all working as expected, and good input, if this explanation was in the first post, it would be easy for you for us to give you right direction, rather suggesting different methods.

Finally the problem is solved good that what is important, if this is solve closed as solved. so it will be usfull for other users in the forum have same issue can refer this thread.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Bryson F · ‎09-12-2022

I have had the same issue before with different model switches. But a rule of thumb i have kept since the first usbflash OS upload is to format the USB to FAT32. Also do not shy away from repeating the same process more than once sometimes the first try wont work, and I recently had this issue today.