3650 switch can not update microcode

[benedicto.rosa]

I was performing an IOS upgrade when we got hit by a major storm. We had a blackout in the middle of the upgrade and lost power. Now the switch just keeps restarting itself. I have provided below the screen shoot of what is going on:

Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1 mismatch: 1

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0], index=0 ...88424 bytes.
Front-end Microcode IMG MGR: Programming device 0...
% Front-end Microcode IMG MGR: address 0, Could not enable sys_poll

% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache
Waiting for 120 seconds for other switches to boot

Can anybody help me with this. I have tried to format the flash and no good. I have tried to reboot from usbflash and nothing. I come to you for guidance and help.

Thank you

[balaji.bandi]

what is the current version, what is the target version you try to upgrade?

[benedicto.rosa]

So the current version was 16.6.8 and going to 16.6.9. As you can see below, there are no traces of 16.6.8, just 16.6.9:

38722 -rw- 2097152 nvram_config
38723 -rw- 2097152 nvram_config_bkup
38724 -rw- 19167872 cat3k_caa-guestshell.16.06.09.SPA.pkg
38725 -rw- 30424184 cat3k_caa-rpbase.16.06.09.SPA.pkg
38726 -rw- 336573048 cat3k_caa-rpcore.16.06.09.SPA.pkg
38727 -rw- 5100160 cat3k_caa-srdriver.16.06.09.SPA.pkg
38728 -rw- 16271996 cat3k_caa-webui.16.06.09.SPA.pkg
38729 -rw- 4742 packages.conf
61953 drwx 4096 .installer
38730 -rw- 407 bootloader_evt_handle.log
69697 drwx 4096 core
77441 drwx 4096 .prst_sync
11 drwx 4096 .rollback_timer
7745 drwx 4096 gs_script
15489 drwx 4096 dc_profile_dir
38731 -rw- 128103 memleak.tcl

Hope this helps.

Thank you

[balaji.bandi]

Is the boots up or keep rebooting?

Looks like you hitting with Bug :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva30394/?rfs=iqvred

Contact TAC for RMA here i guess?

[Leo Laohoo]

Agree with @balaji.bandi and it could be CSCva30394. 

Is the switch boot into ROMMON?  If it is, download any other IOS-XE version, like 16.9.10, and then proceed to load the firmware using emergency-install.  

If the switch boots 16.9.8 successfully, then downgrade to 16.6.10.

If the switch does not boot into ROMMON or does above method does not work, contact Cisco TAC and arrange for an RMA. 

[Mohsin Alam]

Can you post the output of set from ROMMON :

rommon> set 

## Make sure to mark post as helpful, If it resolved your issue. ##

[benedicto.rosa]

switch: set

ABNORMAL_RESET_COUNT=1

ASIC_PCI_RESET=1

AUTOREBOOT_RESTORE=0

BOOT=flash:packages.conf

BOOT_LOADER_UPGRADE_DISABLE=1

BSI=0

CFG_MODEL_NUM=WS-C3650-48FS-E

CLEI_CODE_NUMBER=INM2M00ARA

CSR_PCIERST_DISCONNECTED=yes

DC_COPY=yes

DEFAULT_ROUTER=192.0.2.1

ECI_CODE_NUMBER=471210

IP_ADDR=192.0.2.123/255.255.255.0

LICENSE_BOOT_LEVEL=ipservicesk9,all:ngwc;

MAC_ADDR=C4:F7:D5:DD:B8:80

MANUAL_BOOT=yes

MODEL_NUM=WS-C3650-48PS

MODEL_REVISION_NUM=Q0

MOTHERBOARD_ASSEMBLY_NUM=73-15901-06

MOTHERBOARD_REVISION_NUM=B0

MOTHERBOARD_SERIAL_NUM=FDO23240XZ9

POE1_ASSEMBLY_NUM=73-16439-02

POE1_REVISION_NUM=A0

POE1_SERIAL_NUM=FDO23230130

POE2_ASSEMBLY_NUM=73-16439-02

POE2_REVISION_NUM=A0

POE2_SERIAL_NUM=FDO2323019K

RANDOM_NUM=2024316690

RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin

SWITCH_NUMBER=1

SWITCH_PRIORITY=1

SYSTEM_SERIAL_NUM=FDO2324F2GR

TAN_NUM=800-41231-04

TAN_REVISION_NUMBER=E0

TEMPLATE=advanced

TERMLINES=0

VERSION_ID=V05

[Mohsin Alam]

You need to go into ROMMON and "set BOOT_LOADER_UPGRADE_DISABLE=0"

## Make sure to mark post as helpful, If it resolved your issue. ##