cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9074
Views
10
Helpful
6
Replies

got stuck about recovering IOS for cisco c3850 switch

Johnny Nguyen
Level 1
Level 1

dear all

i have a lot of problems with a switch that was lost IOS autimatically after the power down. i have tried many ways in order to recover it but i have failed all the time. can anyone help me ? below is the error log i got

 

Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@done.
Memory Test Pass!

Base ethernet MAC Address: 18:8b:45:a9:34:80
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 2 seconds....done Initializing Flash.
file name too long

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.

 

thanks for any suggestions

6 Replies 6

Mark Malone
VIP Alumni
VIP Alumni

Hi try follow this should get you back  online :)

 

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html

Recover from a 3850 Series Switch Boot Failure

There are multiple reasons for a 3850 Series Switch boot failure, such as a corrupt boot image, a corrupt packages.conf file, or missing files. This section describes possible recovery methods.

Note: Ensure that you have knowledge of the two possible boot modes, INSTALL and BUNDLE, before you continue.

Standard Recovery Methods

This section describes the two standard methods that are used in order to recover from a Catalyst 3850 Series Switch boot failure.

USB Recovery

The 3850 Series Switches have a USB port on the front that is used for console access. This USB port is also used with flash drives for image backup and recovery.

If you become stuck at the switch: prompt with a corrupt image or .conf file, you can easily boot to a file that is stored on the USB drive or copy an image from the USB to internal flash. Complete these steps in order to recover from the boot failure:

  1. Verify that the flash drive is recognized and the .bin file exists:

    switch: 
    dir usbflash0:

    Directory of usbflash0:/
    74  -rw-  223734376  cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin
  2. Boot to the USB image:

    switch: 
    boot usbflash0:cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

Note: This process boots the switch into BUNDLE mode.

Tip: You can also copy the .bin file from usbflash0: to flash:, and point the boot statement towards internal flash.

Corrupt File Recovery

There are instances when the packages.conf calls files that no longer exist in flash. You can manually boot an image from the switch: prompt file; however, upon reload it calls the packages.conf file again and fails to boot. If this occurs, Cisco recommends that you back up the current packages.conf file and rename it or delete it. This process is mandatory, as the next step fails if a .conf file already exists. Once you extract the .bin file, a new packages.conf file is created. Complete these steps in order to recover from a corrupt packages.conf file:

  1. Once booted (in BUNDLE mode), verify the files in the flash:

    Switch#
    dir flash:

    Directory of flash:/
    15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf
  2. Copy or rename the current packages.conf file:

    Switch#
    cp flash:packages.conf flash:packages.conf.badop

    Destination filename [packages.conf.bad]?
    Copy in progress...C
    1243 bytes copied in 0.140 secs (8879 bytes/sec)

    Switch#
    dir flash:

    Directory of flash:/
    15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf
    15502  -rw-        1243   Aug 1 2013 11:53:51 +00:00  packages.conf.bad
    Switch#
    del flash:packages.conf

    Delete filename [packages.conf]?
    Delete flash:/packages.conf? [confirm]
  3. Expand the bundle in order to create a new packages.conf file:

    Switch#
    software expand running switch 1 to flash:

    Preparing expand operation ...
    [1]: Expanding the running bundle
    [1]: Copying package files
    [1]: Package files copied
    [1]: Finished expanding the running bundle
  4. Verify the boot:

    Switch#
    show boot

    ---------------------------
    Switch 1
    ---------------------------
    Current Boot Variables:
    BOOT variable does not exist

    Boot Variables on next reload:
    BOOT variable = flash:packages.conf;
    Manual Boot = no
    Enable Break = no
  5. Reload the switch:

    Switch#
    reload

    Reload command is being issued on Active unit, this will reload the whole stack
    Proceed with reload? [confirm]

Emergency Recovery

If the previous recovery methods fail, the 3850 Series Switches have a trap door method that you can use in order to recover the system. You must have a terminal that is connected to the management port of the switch that runs a TFTP server. Download a valid image file from CCO and store it in the root of the TFTP server.

It is likely that the switch is stuck at the switch: prompt. However, if you are in a boot loop, you can use the Mode button on the front of the switch in order to break the cycle: hold the button for approximately ten seconds, and the switch breaks the cycle and stops at the switch: prompt.

Complete these steps in order to perform an emergency recovery:

  1. Set the switch IP address:

    switch:  
    set IP_ADDR 192.0.2.123/255.255.255.0
  2. Set the default gateway:

    switch: 
    set DEFAULT_ROUTER 192.0.2.1
  3. Ping the terminal that contains the TFTP server in order to test the connectivity:

    switch: 
    ping 192.0.2.1

    ping 192.0.2.1 with 32 bytes of data ...
    Host 192.0.2.1 is alive.
  4. Verify that the emergency files exist in the switch file system:

    switch: 
    dir sda9:

    Directory of sda9:/

        2  drwx  1024       .
        2  drwx  1024       ..
       11  -rwx  18958824   cat3k_caa-recovery.bin
    36903936 bytes available (20866048 bytes used)
  5. Run the emergency install feature:

    switch: 
    emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.
     SPA.03.03.00.SE.150-1.EZ.bin

    The bootflash will be erased during install operation, continue (y/n)?Y
    Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.
     SPA.03.02.02.SE.150-1.EX2.bin)...
    Reading full image into memory......................done
    Nova Bundle Image
    --------------------------------------
    Kernel Address    : 0x6042f5d8
    Kernel Size       : 0x317ccc/3243212
    Initramfs Address : 0x607472a4
    Initramfs Size    : 0xdc6546/14443846
    Compression Format: .mzip

    Bootable image at @ ram:0x6042f5d8
    Bootable image segment 0 address range [0x81100000, 0x81b80000]
     is in range [0x80180000, 0x90000000].
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
     @@@@@@@@@@@@@@@@@@@
    File "sda9:cat3k_caa-recovery.bin" uncompressed and installed,
     entry point: 0x811060f0
    Loading Linux kernel with entry point 0x811060f0 ...
    Bootloader: Done loading app on core_mask: 0xf

    ### Launching Linux Kernel (flags = 0x5)

    Initiating Emergency Installation of bundle tftp://192.0.2.1/
     cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin

    Downloading bundle tftp://192.0.2.1/ cat3k_caa-universalk9.
     SPA.03.03.00.SE.150-1.EZ.bin...

    Validating bundle tftp://192.0.2.1/ cat3k_caa-universalk9.
     SPA.03.03.00.SE.150-1.EZ.bin...
    Installing bundle tftp://192.0.2.1/ cat3k_caa-universalk9.
     SPA.03.03.00.SE.150-1.EZ.bin...
    Verifying bundle tftp://192.0.2.1/ cat3k_caa-universalk9.
     SPA.03.03.00.SE.150-1.EZ.bin...
    Package cat3k_caa-base.SPA.03.03.00.SE.pkg is Digitally Signed
    Package cat3k_caa-drivers.SPA.03.03.00.SE.pkg is Digitally Signed
    Package cat3k_caa-infra.SPA.03.03.00.SE.pkg is Digitally Signed
    Package cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg is Digitally Signed
    Package cat3k_caa-platform.SPA.03.03.00.SE.pkg is Digitally Signed
    Package cat3k_caa-wcm.SPA.10.0.111.0.pkg is Digitally Signed
    Preparing flash...
    Syncing device...
    Emergency Install successful... Rebooting
    Restarting system.

thanks for your reply i already followed this document but it was unsuccessful. i got this error log and it's impossible to rename file packages.conf in rommon mode because the flash just has read-only permission any suggestions bro ?

 

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin
Reading full image into memory....................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address    : 0x6044b274
Kernel Size       : 0x418331/4293425
Initramfs Address : 0x608635a8
Initramfs Size    : 0xe780c1/15171777
Compression Format: .mzip

Bootable image at @ ram:0x6044b274
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin" uncompressed and installed, entry point: 0x8166a500
Loading Linux kernel with entry point 0x8166a500 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services


FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
Sep 6 18:13:21 %IOSXE-3-PLATFORM: process sysmgr: Service [stack-mgr] pid:[5925] terminated abnormally [11].
Sep 6 18:13:21 %IOSXE-2-PLATFORM: process ng_dumper: Process stack-mgr: terminated abnormally.
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: ===================== Exception Data ====================
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Exception in 'stack-mgr' with Signal=SIGSEGV(11)
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Frame Pointer=0x7F9B4460, Program Counter=0x2BD603D8
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Thread ID=5925: -Traceback= 1#006375bad0c1f9ef5672fa4cefe5c5dd ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC initcii:2CCCD000+F28 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CC
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr:

<Wed Sep  6 18:13:21 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[5925] terminated abnormally [11].
Details:
--------
Service: stack-mgr
Description: Stack Manager
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.06.03E.pkg//usr/binos/bin/stack-mgr

Started at Wed Sep  6 18:12:26 2017 (305531 us)
Stopped at Wed Sep  6 18:13:21 2017 (343002 us)
Uptime: 55 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 5925
Exit code: signal 11

CWD: /var/sysmgr/work


PID: 5925
UUID: 3006
get_local: failed: No such file or directory
/dev/fd/9: line 15:  7658 Segmentation fault      (core dumped) flashutil -c setvar -N ABNORMAL_RESET_COUNT -V ${abnormal_reset_count}

You need to TAC that looks like something in HW is crashing during boot process could need RMA

thanks bro please tell me more. i tried to solve it but always got stuck with stack-mgr in rommom mode. who should i contact ? what should i do ?

FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
Sep 7 19:30:25 %IOSXE-3-PLATFORM: process sysmgr: Service [stackSep 7 19:30:26 %IOSXE-2-PLATFORM: process ng_dumper: Process stack-mgr: terminated abnormally.
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: ===================== Exception Data ====================
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Exception in 'stack-mgr' with Signal=SIGSEGV(11)
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Frame Pointer=0x7FD54490, Program Counter=0x2C86AAD8
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Thread ID=6325: -Traceback= 1#972af6ff8130dc1a2a2858bbb94cad79 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 initcii:2D78C000+F28 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D7
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr:

<Thu Sep  7 19:30:25 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[6325] terminated abnormally [11].
Service: stack-mgr
Description: Stack Manager
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.07.05E.pkg//usr/binos/bin/stack-mgr

Started at Thu Sep  7 19:29:25 2017 (747866 us)
Stopped at Thu Sep  7 19:30:25 2017 (917365 us)
Uptime: 1 minutes 0 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 6325
Exit code: signal 11

CWD: /var/sysmgr/work


PID: 6325
UUID: 3006
get_local: failed: No such file or directory

/dev/fd/9: line 20:  8142 Segmentation fault      (core dumped) flashutil -c setvar -N ABNORMAL_RESET_COUNT -V ${abnormal_reset_count}

This below is a traceback error/bug accomnied by an internal crash file , they can be hardware or software , if you have already tried multiple versions of ios and its still happeneing its most likely hardware fault , only Cisco TAC can tell you exactly what each traceback means , you need to log a cse out with them if its under support

 

-Traceback= 1#972af6ff8130dc1a2a2858bbb94cad79 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 initcii:2D78C000+F28 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D7

oh thank you very much i think only TAC can help me with this case. actually i have tried to solve this problem in more then one week looking for many documents lots and lots of topics forums but i still fail to solve it. if there is no choice i should send it back to the head quater and let them taking warranty. thanks