Impossible to remove file in ROMMON - ISR4331 no command available

[NicolasDemonty]

Hi

we are facing a memory issue and the bootflash: is totally full. Therefore the ISR4331 running on version 16.12.6 is rebooting in a loop. 

From rommon I'm not able to clean flash: as there isn't any commany "del" "delete" "erase" "format" or even "xmodem". This is also not an option to unmount the device and remove any hardware.

Does anyone have any idea how files can be deleted from ROMMON ?

thanks in advance

[Leo Laohoo]

Boot the firmware using a USB thumb drive.

[NicolasDemonty]

Hello

thanks for the reply but this is not an option as the device is 4000km from here

any info about the missiing ROMMON commands ? Is there any alternative to the missing commands ?

thanks

[Leo Laohoo]

Is this router running SD-WAN firmware?

[balaji.bandi]

post the complete output from console, so we can undertand better to suggest you right direction,

[NicolasDemonty]

Hi,

in summary 

Mar 21 08:27:47.160: %BOOT-0-BOOT_NOSPACE: R0/0: WARNING: No space on bootflash /dev/bootflash5 for packages, using system memory! Delete files from bootflash to free memory and reload the system; otherwise, system behavior will be unpredictable due to high memory utilization.

Mar 21 08:28:47.650: %FLASH_CHECK-3-DISK_QUOTA: R0/0: flash_check: Flash disk quota exceeded [free space is 0 kB] - Please clean up files on bootflash.

then the ISR reboots continuously and only way it the reach the ROMMON by breaking the sequence. However in ROMMON I can't manage anything in the flash. All commands for deleting a file don't exist anymore. I would need a way to delete files but by beeing in ROMMON. Here below the otpions I have

rommon 9 > help
alias set and display aliases command
boot boot up an external process
confreg configuration register utility
dev list the device table
dir list files in file system
help monitor builtin command help
history monitor command history
meminfo main memory information
repeat repeat a monitor command
reset system reset
set display the monitor variables
showmon display currently selected ROM monitor
sync write monitor environment to NVRAM
token display board's unique token identifier
unalias unset an alias
unset unset a monitor variable

[balaji.bandi]

Looks like this may be bug :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvs81967

%FLASH_CHECK-3-DISK_QUOTA: R0/0: flash_check: Flash disk quota exceeded [free space is 0 kB] - Please clean up files on bootflash.

it show as 0 space here.

If you have USB - copy image to USB, and boot from usb, and clear some files from bootflash:

https://www.cisco.com/c/en/us/support/docs/routers/4000-series-integrated-services-routers/200678-Troubleshoot-Cisco-4000-Series-ISR-Stuck.html

[NicolasDemonty]

Hi 

Thanks

However this doesn't help as I can't count on local support. So USB key option is not possible.

Has anyone an idea if there is still commands in ROMMON to delete file. Why have those commands disappeared ?

[balaji.bandi]

try :

dir flash:  (what files you see)

use delete flash:/xxxxx    (xxx file  to delete )

Why have those commands disappeared ?

what command ?

[NicolasDemonty]

rommon 2 > delete flash:isr4400-universalk9.16.12.06.SPA.bin
monitor: command "delete" not found

delete doesn't exist in ROMMON, this is my issue, there isn't possibility to clean flash: from ROMMON. On other platform "delete" exists but not here on this ISR4431 (sorry I mentionned 4331 before but this is 4431)

[balaji.bandi]

what is the output

dir flash:

[NicolasDemonty]

rommon 5 > dir flash:

0 0 drw- .installer
0 10366 -rw- saved_comfig-Mar--8-09-48-04.083-34
0 35 -rw- pnp-tech-time
0 166342 -rw- pnp-tech-discovery-summary
0 0 drw- core
0 0 drw- .prst_sync
0 0 drw- .rollback_timer
0 0 drw- gs_script
0 0 drw- tracelogs
0 0 drw- .dbpersist
0 0 drw- virtual-instance
0 0 drw- vman_fdb
0 30 -rw- throughput_monitor_params
0 0 drw- onep
0 642638316 -rw- isr4400-universalk9.16.12.06.SPA.bin
0 132108 -rw- memleak.tcl
0 0 drw- license_evlog
0 10366 -rw- saved_comfig-Mar--8-10-30-41.607-36
0 0 drw- lost+found
0 0 -rw- mode_event_log
0 0 -rw- ios_core.p7b
0 10323 -rw- saved_comfig-Mar--7-10-02-21.986-33
0 10366 -rw- saved_comfig-Mar--8-10-18-48.229-35
0 704607430 -rw- isr4400-universalk9.17.03.04a.SPA.bin
0 10389 -rw- saved_comfig-Mar-15-11-33-21.634-37
0 0 -rw- trustidrootx3_ca.ca

[Leo Laohoo]

---

@NicolasDemonty wrote:

isr4400-universalk9.17.03.04a.SPA.bin

---

If the router was operating in 16.12.X, then delete the file above.  
It should give you enough space to boot the router and do a clean-up. 

When the router boots up, check the contents of the "core" sub-directory:  dir flash:core

[Leo Laohoo]

Oh I am such an id10t!.  
Yeah, no "delete" command in ROMMON.  <FACEPALM>

Alright, here's how it goes: Post the entire boot-up process.  I have seen this before and it is a bug with the ROMMON with the early generation, early release of routers.  The ROMMON bug means some of the "usual" ROMMON commands are missing.

Now, I have good idea why the router is in a boot-crash-loop:  The router is upgrading from 16.X.X to 17.X.X, right? 

The bootflash does not contain any ROMMON upgrade files.  This means no one has made any attempt to upgrade the ROMMON.  And this is why the router is in a boot-crash-loop.  

This is why it is recommended to upgrade the ROMMON but, hey, who actually reads the Release Notes, right?

[balaji.bandi]

agreed with the suggestion, biggest space will be freed if you delete :  ( are you trying to upgrade 17.3 ?)

0 704607430 -rw- isr4400-universalk9.17.03.04a.SPA.bin