3850 stuck in rommon; can't install IOS due to low flash storage

[bdebriae]

I have (4) WS-C3850-48P running IOS ver 16.6.8 in INSTALL mode that are stacked. We had an unexpected power outage and upon restoration, one of the switches didn't come back up. I removed power, disconnected it from the stack and after powering back up, I'm in rommon. For one reason or another, the packages.conf file is missing from flash, so it can't boot. 

I'm running into issues when attempting to recover the switch. I have the .bin file on a USB drive connected to the switch. 

-When attempting an emergency-install, it fails due to low flash available space. 

-When I attempt to boot from usb, it fails due to low flash available space. 

-I can't delete flash from rommon as the file system is read only. 

How can I clear enough flash space without being able to boot the IOS?

switch: dir

List of filesystems currently registered:

xmodem[0]: (read-only)
null[1]: (read-write)
bs[3]: (read-only)
flash[18]: (read-only)
tftp[19]: (read-write)
ftp[20]: (read-only)
http[21]: (read-only)
usbflash0[22]: (read-write)
usbflash0[23]: (read-write)

switch: dir flash:
Directory of flash:/

15521 drwx 4096 .
2 drwx 4096 ..
15533 -rwx 2097152 nvram_config
15523 -rw- 2097152 nvram_config_bkup
15526 -rw- 35 pnp-tech-time
15527 -rw- 94104 pnp-tech-discovery-summary
15522 -rwx 616 vlan.dat
38804 -rw- 19167872 cat3k_caa-guestshell.16.06.08.SPA.pkg
38808 -rw- 30417560 cat3k_caa-rpbase.16.06.08.SPA.pkg
15531 -rw- 1776 bootloader_evt_handle.log
46562 drwx 4096 core
54321 drwx 4096 .prst_sync
69841 drwx 4096 .rollback_timer
77601 drwx 4096 dc_profile_dir
77603 drwx 4096 gs_script
15532 -rw- 128103 memleak.tcl
85361 drwx 4096 .installer
38805 -rw- 336503416 cat3k_caa-rpcore.16.06.08.SPA.pkg
15535 -rw- 6925 dfr_config
15541 -rw- 4742 16.06.06.conf
38807 -rw- 5091968 cat3k_caa-srdriver.16.06.08.SPA.pkg
38806 -rw- 16271996 cat3k_caa-webui.16.06.08.SPA.pkg
15537 drwx 4096 tech_support
15525 -rw- 76 boothelper.old
15571 drwx 4096 .dbpersist
15572 -rw- 0 rdope_out.txt
15573 -rw- 545 rdope.log
15574 drwx 4096 onep

1208156160 bytes available (441929728 bytes used)

switch: dir usbflash0:
Directory of usbflash0:/

4 -rw- 407458043 cat3k_caa-universalk9.16.06.08.SPA.bin

30362238976 bytes available (386187264 bytes used)

switch: emergency-install usbflash0:cat3k_caa-universalk9.16.06.08.SPA.bin
The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery (usbflash0:cat3k_caa-universalk9.16.06.08.SPA.bin)...
Not enough free memory for entire image
Not enough free memory for entire image
sda9:cat3k_caa-recovery.bin: bad/non-bootable bundle file

Error loading "sda9:cat3k_caa-recovery.bin"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

Cannot access recovery file (sda9:cat3k_caa-recovery.bin and sda9:c3850-recovery.bin)
Call Cisco Systems Technical Support
usage: emergency-install url://<url>

switch: boot usbflash0:cat3k_caa-universalk9.16.06.08.SPA.bin
Not enough free memory for entire image
Not enough free memory for entire image
usbflash0:cat3k_caa-universalk9.16.06.08.SPA.bin: bad/non-bootable bundle file

Error loading "usbflash0:cat3k_caa-universalk9.16.06.08.SPA.bin"

Interrupt within 5 seconds to abort boot process.
Boot process failed...

switch: ?
? -- Present list of available commands
arp -- Show arp table or arp-resolve an address. -f to flush
boot -- Load and boot an executable image
cat -- Concatenate (type) file(s)
copy -- Copy a file
delete -- Delete file(s)
dig -- Lookup a name via a remote dns server
dir -- List files in directories
emergency-install -- Initiate Disaster Recovery
flash_init -- Initialize filesystem(s)
format -- Format a filesystem
fsck -- Check filesystem consistency
help -- Present list of available commands
mgmt_init -- initialize management port
mkdir -- Create dir(s)
mlog -- In-memory log control
more -- Concatenate (display) file(s)
net-debug -- Show/change network debug val
net-dhcp -- Initiate DHCP request for remote configuration
net-show -- Show network parameters
net-tcp-bufs -- see/set tcp number of buffers
net-tcp-mss -- see/set tcp mss
net6-dhcp -- test dhcpv6
net6-show -- Display ipv6 info
ping4 -- Send ICMP ECHO_REQUEST packets to a network host
ping6 -- Ping v6 host
readrtc -- Read RTC
rename -- Rename a file
reset -- Reset the system
rmdir -- Delete empty dir(s)
set -- Set or display environment variables
set_bs -- Set attributes on a boot sector filesystem
set_param -- Set system parameters in flash
type -- Concatenate (type) file(s)
unset -- Unset one or more environment variables
version -- Display boot loader version
writertc -- Write RTC

switch: flash_init
Initializing Flash...
...The flash is already initialized.

switch: delete flash:cat3k_caa-rpcore.16.06.08.SPA.pkg
Are you sure you want to delete "flash:cat3k_caa-rpcore.16.06.08.SPA.pkg" (y/n)?y
File "flash:cat3k_caa-rpcore.16.06.08.SPA.pkg" not deleted -- read only file system

[Kasun Bandara]

try this

https://community.cisco.com/t5/switching/changeing-flash-from-read-only-to-read-write/td-p/3739978

[bdebriae]

Thanks. Load_helper is not a recognized command. I tried both commands below followed by a boot command, and both attempted to load packages.conf which is not in flash and ended up back in rommon.
Set BYPASS_STARTUP_CONFIG 1
Set SWITCH_IGNORE_STARTUP_CFG 1

[marce1000]

                   - What about flash_init     , this command being mentioned first

 M.

[bdebriae]

I had already initialized flash successfully.

[marce1000]

                 >...I had already initialized flash successfully.

                Did  it help to get read/write access again ?

 M.

[bdebriae]

Flash is read only, before and initializing flash. I believe I may have solved it, will update shortly.

[marce1000]

 - You may need to initialize the flash , check this thread for how to (s) : https://community.cisco.com/t5/switching/changeing-flash-from-read-only-to-read-write/td-p/3739978

 M.

[bdebriae]

Thanks, but flash had already been initialized.

[Leo Laohoo]

---

@bdebriae wrote:

sda9:cat3k_caa-recovery.bin: bad/non-bootable bundle file
Error loading "sda9:cat3k_caa-recovery.bin"

---

RMA the switch.  The Golden Capsule ROMMON image is corrupted.  

[bdebriae · Accepted solution]

I ended up formatting the USB drive once more, re-downloaded the IOS and transferred to the USB once more. After that emergency-install worked. The IOS must have gotten corrupt during the first download. Guess I should have verified md5 first.