Booting issue with Cat3850 while upgrading to 16.6.8

[alanvthomas1991@gmail.com]

While upgrading a c3850 switch from 16.3.5 to 16.6.8 in the install mode, i am came across the following issue:

Booting...cf_identify_drive: cf_waitfor(!(BUSY | ERROR)): timed out.
Boot bus: Failed to detect any IDE storage!

Interface GE 0 link down***ERROR: PHY link is down
flash:packages.conf: no such device
Cannot open file flash:packages.conf

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.

switch: dir

List of filesystems currently registered:

xmodem[0]: (read-only)
null[1]: (read-write)
bs[3]: (read-only)
tftp[10]: (read-write)

I would like to know the following:

- is the switch recoverable?

- if the switch is recoverable can i install the switch using xmodem or usb?

- What caused this issue? I had checked the Md5 value of the image installed on the switch and compared it to the one on the cisco website and they were both found to be same. How do i prevent this issue in the future as we have more switches which will be upgraded to 16.6.8?

[marce1000]

  - Looks like a serious hardware fault , last resort : power cycle and or factory reset , if that does not help , have the device replaced (RMA).

 M.

[Reza Sharifi]

What upgrade procedure did you use? See below documents:

https://community.cisco.com/t5/networking-knowledge-base/3850-switch-ios-xe-upgrade-detailed-standalone/ta-p/3138609

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html

HTH

[alanvthomas1991@gmail.com]

I have used the standalone Install mode. Copied the image to the switch then followed the process to upgrade the switch in the Install mode.

I will try the emmergency recovery process to see if it helps that's the last step pending. I tried to load the image using Xmodem and that failed:

switch: flash_init
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.

switch: load_helper
Unknown cmd: load_helper

switch: dir

List of filesystems currently registered:

xmodem[0]: (read-only)
null[1]: (read-write)
bs[3]: (read-only)
tftp[10]: (read-write)

switch: dir flash:
unable to stat flash:/: no such device

There is no flash in the switch

[alanvthomas1991@gmail.com]

There is no emergency directory as well:

switch: dir sda9:
unable to stat sda9:/: no such device

switch:

[Reza Sharifi]

Try the USB recovery:

Standard Recovery Methods

This section describes the two standard methods that are used in order to recover from a Catalyst 3850 Series Switch boot failure.

USB Recovery

The 3850 Series Switches have a USB port on the front that is used for console access. This USB port is also used with flash drives for image backup and recovery.

If stuck at the switch: prompt with a corrupt image or .conf file, boot to a file that is stored on the USB drive or copy an image from the USB to internal flash. Complete these steps in order to recover from the boot failure:

[alanvthomas1991@gmail.com]

I will try to boot it from the USB and see if this helps. It needs to use the image stored in the USB, cannot save it to the flash in the switch as the flash does not exist.

[Reza Sharifi]

cannot save it to the flash in the switch as the flash does not exist.

Every switch has a flash. I know for sure that the 3850s come with flash. So, it is not that there is no flash, it may be that there is no room in the flash. If you issue "dir" or "sh flash" you will see the flash with all the files in it, how much space is taken, and how much space is left.

HTH

[Leo Laohoo]

When in ROMMON, post the complete output to the following commands: 

dir flash:
set

Next, have a look at this:  Cisco 3850: IOS-XE/Firmware Upgrade (Install Mode) and scroll down to Section 4:  Emergency-Install.  

[alanvthomas1991@gmail.com]

switch: dir

List of filesystems currently registered:

xmodem[0]: (read-only)
null[1]: (read-write)
bs[3]: (read-only)
tftp[10]: (read-write)

switch: dir flash:
unable to stat flash:/: no such device

switch: set
ABNORMAL_RESET_COUNT=0
ASIC_PCI_RESET=1
BAUD=115200
BOOT=flash:packages.conf;
BOOT_LOADER_UPGRADE_DISABLE=1
BSI=0
CFG_MODEL_NUM=WS-C3850-12X48U-L
CLEI_CODE_NUMBER=IPM8Y10ARB
CSR_PCIERST_DISCONNECTED=yes
DC_COPY=yes
DEFAULT_ROUTER=10.1.1.5
D_STACK_DOMAIN_NUM=1
ECI_CODE_NUMBER=470242
IP_ADDR=10.1.1.10/255.255.255.0
LICENSE_BOOT_LEVEL=lanbasek9,all:ngwc;
MAC_ADDR=68:2C:7B:B0:9D:00
MANUAL_BOOT=no
MODEL_NUM=WS-C3850-12X48U
MODEL_REVISION_NUM=M0
MOTHERBOARD_ASSEMBLY_NUM=73-15755-08
MOTHERBOARD_REVISION_NUM=B0
MOTHERBOARD_SERIAL_NUM=FOC22056CQY
POE1_ASSEMBLY_NUM=73-16123-03
POE1_REVISION_NUM=A0
POE1_SERIAL_NUM=FOC22060N7S
POE2_ASSEMBLY_NUM=73-16123-03
POE2_REVISION_NUM=A0
POE2_SERIAL_NUM=FOC22060MH8
RANDOM_NUM=532114772
RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin
RET_2_RCALTS=1658444420
RET_2_RTS=18:00:20 CDT Thu Jul 21 2022
STKPWR_ASSEMBLY_NUM=73-11956-08
STKPWR_REVISION_NUM=B0
STKPWR_SERIAL_NUM=FOC22061P73
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=FOC2208U038
TAN_NUM=800-41071-02
TAN_REVISION_NUMBER=F0
TEMPLATE=advanced
TERMLINES=0
USB_ASSEMBLY_NUM=73-16167-02
USB_REVISION_NUM=A0
USB_SERIAL_NUM=FOC22062XA0
VERSION_ID=V02

Spoke with TAC they suggested to RMA the switch as there is no flash in the switch.

The reason we wanted to upgrade the switch was the switch would randomly reboot so we were not sure what caused the issue and decided we would upgrade the switch from 16.3.5 to 16.6.8. During the upgrade the switch did not boot back up and when we consoled in we found there is no flash in the switch.

We are not sure what caused the issue, as earlier we had successfully upgraded the same model of switches from code 3.x.x.x to 16.6.8 without any issues in the bundle mode, so we are just wondering what caused the failure here. The only difference is the affected switch was in Install mode.