Cisco 3850 stuck in rommon flash read only

[MIKE CLARK]

Hi 

we have a 3850 which fails to boot I can see the files on the flash however it is read only so can't make any changes. If I boot from USB switch performs firmware upgrades but then fails to boot have tried booting numerous times from USB. Tried emergency load from tftp server first part loads agreed to format flash switch waits to download files 2nd time but nothing is requested from tftp server. Load of hyroglifics in the set statement I tried ignoring the startup configuration but no difference.

how do I delete or format the flash so I can start from scratch

[Reza Sharifi]

how do I delete or format the flash so I can start from scratch

try:

format flash: 

you have options to format fat16 or 32.

HTH

[MIKE CLARK]

Reza thanks for replying, the flash: is Read-Only so you cant format, change or delete anything I think there is a hardware fault as it makes no difference even if the Emergency-install procedure is followed which appears to format or replace the files on the flash:

[Reza Sharifi]

Hi Mike,

It could be hardware problem. Try looking at the file by issuing this command:

"sh file systems" under the flag section, does it say "r" or "rw"?

HTH

[Leo Laohoo]

Did you try to DOWNGRADE the IOS of the switch when this happens?  There is an IOS bug affecting a large quantity of 3850.  These switches are shipped with Denali and they will go into ROMmon when they are downgraded to 3.6.X and (earlier).  The switch, however, will boot 3.7.X properly.  

[MIKE CLARK]

Hi Leo,

Not sure what happened before I got hold of it, but I tried with Release Denali-16.3.1 16.03.01a

the firmware was upgraded then I got these messages and the box reboots.

May 22 12:46:37.258 R0/0: %PMAN-3-PROCHOLDDOWN: The process sif_mgr has been helddown (rc 139)

May 22 12:46:37.825 R0/0: %PMAN-0-PROCFAILCRIT: A critical process sif_mgr has failed (rc 139)

May 22 12:46:38.294 R0/0: %PMAN-3-RELOAD_RP: Reloading: Switch will be reloaded

Chassis 0 reloading, reason - Non participant detected
May 22 12:46:46.922 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested

May 22 12:46:53.048 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code

[Leo Laohoo]

Mike, 

Please console into the switch and reboot.  We want to see what the switch is doing and where it's failing.

[MIKE CLARK]

Hi Leo,

This is an example using the last IOS.

Cisco IOS Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version Denali 16.2.1, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Sun 27-Mar-16 21:47 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2016 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

Error: USB Token Service is not available.

Exception to IOS Thread:
Frame pointer 0xFF99ED81A0, PC = 0xFFECDDAE7C

UNIX-EXT-SIGNAL: Segmentation fault(11), Process = Init
-Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4  c:FFECD2E000+ACE7C flash:FFEE0FD000+3EE0 flash:FFEE0FD000+248C :AAB9E47000+374A610 :AAB9E47000+5FFACA4 :AAB9E47000+3DDE578 :AAB9E47000+3F578F4 :AAB9E47000+3F57DC0 :AAB9E47000+5F95648

Fastpath Thread backtrace:
-Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4  pthread:FFECCBD000+9264 pthread:FFECCBD000+E68C iosd_ngwc_unix:FFECF15000+31420 :AAB9E47000+145F7B4 iosd_ngwc_unix:FFECF15000+23CAC prelib:FFEF484000+2CDC pthread:FFECCBD000+C0AC c:FFECD2E000+10E92C

Auxiliary Thread backtrace:
-Traceback= 1#4e77356e181cb88f5db27cbef08e5fd4 

$0 : 0x00000000, AT : 0xFFEE121E20, v0 : 0x00000000, v1 : 0x00000008
a0 : 0x00000000, a1 : 0x00000000, a2 : 0x00000000, a3 : 0x8080808080808080
t0 : 0xFEFEFEFEFEFEFEFF, t1 : 0x5922C4BD, t2 : 0x2ACBCF80, t3 : 0x5922C4BD
t4 : 0x0206CC80, t5 : 0x5922C4BF, t6 : 0x00000000, t7 : 0x00000000
s0 : 0x00000000, s1 : 0xAACD9CC0A0, s2 : 0x00010000, s3 : 0x0000FFFC
s4 : 0xFFEE0FD000, s5 : 0xFFEE11D000, s6 : 0xFFEE11D000, s7 : 0x0000002B
t8 : 0x00000038, t9 : 0xFFECDDAE00, k0 : 0xFF99ED8158, k1 : 0x00000000
gp : 0xFFEE121E20, sp : 0xFF99ED81A0, s8 : 0xFFEE121E20, ra : 0xFFEE100EE0
EPC : 0xFFECDDAE7C,  SREG : 0x00000000 Cause : 0xFFFFFFFF
Error EPC : 0x00000000, BADVADDR = 0x00000000

Crashinfo file not found.

Buffered messages:
Queued messages:
*May 22 11:00:16.425: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*May 22 11:00:06.043: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
IOSD RP: 2 messages not written to btrace log file
Chassis 1 reloading, reason - Reload command
May 22 11:00:45.096 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
May 22 11:00:51.292 R0/0: %PMAN-3-PROCHOLDDOWN: The process hman has been helddown (rc 134)
May 22 11:02:11.968 R0/0: %PMAN-3-PROCHOLDDOWN: The process linux_iosd-image has been helddown (rc 139)
May 22 11:02:16.300 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code

reboot: Restarting system

[jasonfrank2010]

I see the similar thing on my switch, and I don't have any further information than what is posted here.  Was there ever a fix to this?

Thank you

J

[AsimHafeez1]

Did anyone find a solution to this problem ?. I tried many methods to solve this but it didn`t work. And I don know why it booted one time from usb successfully but after reload I couldn`t boot from usb again facing the same problem mentioned above.

[pcareccia]

once you boot from usb or tftp you are supposed to copy the file into flash and fix the problem from there, booting from usb or tftp is only to get you into the switch so you can copy the necessary files etc

[eugeneg]

I thought I'd share the solution I found to this issue - copy the binary to a usbkey and slot it into the front panel USB port.

At the ROMMON switch: prompt issue the command:

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

(replace cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin with the name of your binary)

Once the switch has booted, copy the binary to flash

Switch# copy usbflash0:cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin flash:

Don't forget to set the boot image

Switch# conf t

Switch(config)# boot system flash:cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

hopefully this will solve the issue

[ahmedjamil86]

can you please also brief how to set the binary .
i am facing the same issue and notable to recover my 3850 switch .
i even try to boot it from usbflash0: but still i crushed during startup

Bootable image at @ ram:0x53430388
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816903a0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)


Final tar file: mcu_ucode_bundle_6_2_0.tar

Both links down, not waiting for other switches
Switch number is 1

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706



Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.7, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Fri 03-Aug-18 23:49 by mcpre



Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.




Exception to IOS Thread:
Frame pointer 0xFF9FC341A0, PC = 0xFFF0FEC1AC

UNIX-EXT-SIGNAL: Segmentation fault(11), Process = Init
-Traceback= 1#a6413825815c10fbf9c794e5e3351ef1 iosd_ngwc_unix:FFF1127000+32A10 c:FFF0F40000+AC1AE flash:FFF3BF7000+4280 flash:FFF3BF7000+2824 :AABA708000+3B800A0 :AABA708000+662D844 :AABA708000+425F268 :AABA708000+43D9804 :AABA708000+43D9CD0

Fastpath Thread backtrace:
-Traceback= 1#a6413825815c10fbf9c794e5e3351ef1 pthread:FFF0B50000+9266 pthread:FFF0B50000+E68C iosd_ngwc_unix:FFF1127000+33670 :AABA708000+1753FE4 iosd_ngwc_unix:FFF1127000+248B4 prelib:FFF50D2000+2E94 pthread:FFF0B50000+C0AC c:FFF0F40000+10DC4C

Auxiliary Thread backtrace:
-Traceback= 1#a6413825815c10fbf9c794e5e3351ef1 pthread:FFF0B50000+11176 iosd_ngwc_unix:FFF1127000+38544 prelib:FFF50D2000+2E94 pthread:FFF0B50000+C0AC c:FFF0F40000+10DC4C

$0 : 0x00000000, AT : 0xFFF3C1C870, v0 : 0x00000000, v1 : 0x00000008
a0 : 0x00000000, a1 : 0x00000000, a2 : 0x00000000, a3 : 0x8080808080808080
t0 : 0xFEFEFEFEFEFEFEFF, t1 : 0x5DFE12C7, t2 : 0x34DE8BC0, t3 : 0x5DFE12C7
t4 : 0x2D3D6BC0, t5 : 0x5DFE12C8, t6 : 0x00000000, t7 : 0x00000000
s0 : 0x00000000, s1 : 0xAAD1C8A188, s2 : 0x00010000, s3 : 0x0000FFFC
s4 : 0xFFF3BF7000, s5 : 0xFFF3C17000, s6 : 0xFFF3C17000, s7 : 0x0000002A
t8 : 0x00000038, t9 : 0xFFF0FEC130, k0 : 0xFF9FC34158, k1 : 0x00000000
gp : 0xFFF3C1C870, sp : 0xFF9FC341A0, s8 : 0xFFF3C1C870, ra : 0xFFF3BFB280
EPC : 0xFFF0FEC1AC, SREG : 0x00000000 Cause : 0xFFFFFFFF
Error EPC : 0x00000000, BADVADDR = 0x00000000

Crashinfo file not found.

Buffered messages:
Queued messages:
*Dec 21 12:40:41.884: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Dec 21 12:40:37.932: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*Dec 21 12:40:39.506: %IOSD_INFRA-6-IFS_DEVICE_OIR: Device usbflash0 added
IOSD RP: 3 messages not written to btrace log fileDec 21 12:42:30.060 R0/0: %PMAN-3-PROCHOLDDOWN: The process linux_iosd-image has been helddown (rc 139)
Dec 21 12:42:30.729 R0/0: %PMAN-0-PROCFAILCRIT: A critical process linux_iosd_image has failed (rc 139)
Dec 21 12:42:31.347 R0/0: %PMAN-3-RELOAD_SYSTEM: Reloading: Peer switch is not standby ready. System will be reloaded

Chassis 1 reloading, reason - Reload command
Dec 21 12:42:40.384 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
Dec 21 12:42:47.227 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code
Dec 21 12:43:16.852 R0/0: %PMAN-3-PROCESS_NOTIFICATION: System report /crashinfo/system-report_1_20191221-124257-Universal.tar.gz (size: 18076 KB) generated

octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system

[MerissaLong06411]

I'm a little late but I recently ran into the same issue and was able to resolve it. I had to download the .bin file to a usb and then while in rommon mode confirm the .bin file is showing on the usb, dir usbflash0:

then use the command emergency-install usbflash0:/cat3k_caa-universalk9.16.03.09.SPA.bin or whatever .bin file you're using and it will override the current flash and delete everything not being used. Just be sure to update your boot var to package.conf once it's booted up and you should be good!

[dennishyde]

I've had the same problems as above.  Basically can't copy a new IOS into flash:.  I've booted from USB and toward the end of the process the switch reboots and comes up with nothing in flash or misconfigured.  Back to Switch: prompt.  Will try the emergency install but can't copy from xmodem, usbflash0, or SCP without the read only error.  Checked and all files in Flash: are RW.