get trouble with cisco 3850 switch

Johnny Nguyen · ‎09-05-2017

dear all

i have had a problem with a cisco 3850 switch, actually after the power down the switch lost IOS automatically and i have tried many methods and many ways ( using an usb stick, tftp) in order to recover its IOS but failed all the time below is the error log

FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
MCU type = 0 and image name is main_mcu_ucode_bundle_6_2_0.tar
Final tar file: main_mcu_ucode_bundle_6_2_0.tar
Sep 5 22:42:29 %IOSXE-3-PLATFORM: process sysmgr: Service [stackSep 5 22:42:29 %IOSXE-2-PLATFORM: process ng_dumper: Process stack-m        gr: terminated abnormally.
Sep 5 22:42:30 %IOSXE-2-PLATFORM: process stack-mgr: ===================== Exception Data ====================
Sep 5 22:42:30 %IOSXE-2-PLATFORM: process stack-mgr: Exception in 'stack-mgr' with Signal=SIGSEGV(11)
Sep 5 22:42:30 %IOSXE-2-PLATFORM: process stack-mgr: Frame Pointer=0x7F8EEB40, Program Counter=0x2C7003D8
Sep 5 22:42:30 %IOSXE-2-PLATFORM: process stack-mgr: Thread ID=5929: -Traceback= 1#56f6b7f0279a368c4fa9294ffc3c2696 ld:2B82C000+AE60         ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 ld:2B82C000+        AE60 ld:2B82C000+AE60 ld:2B82C000+AE60 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 c        rashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000        +5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D        681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 crashinfo:2D681000+5A8 initcii:2D66D000+F28 c:2C7E5A78+8F50 c:2C7E5A78+8F50         c:2C7E5A78+8F50 c:2C7E5A78+8F50 c:2C7E5A78+8F50 c:2C7E5A78+8F50 c:2C7E5A78+8F50 c:2C7E5A78+8F50 c:2C7E5A78+8F50 initcii:2D66D000+0         initcii:2D66D000+0 initcii:2D66D000+0 initcii:2D66D000+0 initcii:2D66D000+0 initcii:2D66D000+0 initcii:2D6
Sep 5 22:42:30 %IOSXE-2-PLATFORM: process stack-mgr:

<Tue Sep 5 22:42:29 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[5929] terminated abnormally [1        1].
Service: stack-mgr
Description: Stack Manager
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.06.06E.pkg//usr/binos/bin/stack-mgr

Started at Tue Sep 5 22:41:33 2017 (353686 us)
Stopped at Tue Sep 5 22:42:29 2017 (440055 us)
Uptime: 56 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 5929
Exit code: signal 11

CWD: /var/sysmgr/work

PID: 5929
UUID: 3006
get_local: failed: No such file or directory
/dev/fd/9: line 15: 7682 Segmentation fault      (core dumped) flashutil -c setvar -N ABNORMAL_RESET_COUNT -V ${abnormal_reset_coun        t}

can anyone help me ? i'm lost to find out what problems really are

Austin Sabio · ‎09-06-2017

It looks like you are hitting this bug CSCve02138. I would recommend to usb boot then upgrade/downgrade to recommended ios version based on the switch platform. Side note, usb format needs to be FAT16 or FAT32 as "the default format is FAT16, while FAT32 format is also supported.".

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve02138/?referring_site=bugquickviewredir

Hope this helps. Good luck!

Johnny Nguyen · ‎09-06-2017

thanks a lot for your help. actually i have tried to boot from usbflash but i got this error

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin
Reading full image into memory....................................................................................................................................................................................................................................................................................................done
Bundle Image
--------------------------------------
Kernel Address    : 0x6044b274
Kernel Size       : 0x418331/4293425
Initramfs Address : 0x608635a8
Initramfs Size    : 0xe780c1/15171777
Compression Format: .mzip

Bootable image at @ ram:0x6044b274
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "usbflash0:cat3k_caa-universalk9.SPA.03.06.03.E.152-2.E3.bin" uncompressed and installed, entry point: 0x8166a500
Loading Linux kernel with entry point 0x8166a500 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed
Starting System Services

FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
Sep 6 18:13:21 %IOSXE-3-PLATFORM: process sysmgr: Service [stack-mgr] pid:[5925] terminated abnormally [11].
Sep 6 18:13:21 %IOSXE-2-PLATFORM: process ng_dumper: Process stack-mgr: terminated abnormally.
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: ===================== Exception Data ====================
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Exception in 'stack-mgr' with Signal=SIGSEGV(11)
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Frame Pointer=0x7F9B4460, Program Counter=0x2BD603D8
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr: Thread ID=5925: -Traceback= 1#006375bad0c1f9ef5672fa4cefe5c5dd ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 ld:2AE86000+AE60 crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC crashinfo:2CCE1000+5AC initcii:2CCCD000+F28 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 c:2BE45A78+8F50 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CCCD000+0 initcii:2CC
Sep 6 18:13:22 %IOSXE-2-PLATFORM: process stack-mgr:

<Wed Sep 6 18:13:21 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[5925] terminated abnormally [11].
Details:
--------
Service: stack-mgr
Description: Stack Manager
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.06.03E.pkg//usr/binos/bin/stack-mgr

Started at Wed Sep 6 18:12:26 2017 (305531 us)
Stopped at Wed Sep 6 18:13:21 2017 (343002 us)
Uptime: 55 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 5925
Exit code: signal 11

CWD: /var/sysmgr/work

PID: 5925
UUID: 3006
get_local: failed: No such file or directory
/dev/fd/9: line 15: 7658 Segmentation fault      (core dumped) flashutil -c setvar -N ABNORMAL_RESET_COUNT -V ${abnormal_reset_count}

have you ever had this problem ? any suggestions bro

Julio E. Moisa · ‎09-06-2017

Hi Johnny

It could be a bug but also please check this link:

http://jamsudsanusi.net/2015/02/recovering-catalyst-3850-boot-loop-ios-upgrade/

Hope it is useful

:-)

>> Marcar como útil o contestado, si la respuesta resolvió la duda, esto ayuda a futuras consultas de otros miembros de la comunidad. <<

Johnny Nguyen · ‎09-06-2017

thanks a lot but it didn't work because the switch is now in rommon mode and the flash is read-only permission i can't rename the file packages.conf so it's failed any suggestions bro ?

Austin Sabio · ‎09-06-2017

"<Wed Sep 6 18:13:21 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[5925] terminated abnormally [11]."

It's failing at the stack manager boot.

Here's a cisco documenation on how to troubleshoot it.

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/201070-Troubleshooting-3650-3850-reloads-by-sta.html

However, I would recommend to try different code -downgrade if needed to a stable version.

Johnny Nguyen · ‎09-06-2017

many thanks for your help

i read this article but can't find out the method to fix it although you show me a clearly problem. perhaps it's because the service stack_mgr doesn't work and the switch always stops at rommon mode. please help me

Johnny Nguyen · ‎09-06-2017

i also try to use emergency-install in order to downgrade IOS version but i got another error log. the following error log

switch: emergency-install usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin
The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery (usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin)...
Reading full image into memory......................done
Bundle Image
--------------------------------------
Kernel Address    : 0x6044a5d8
Kernel Size       : 0x31794f/3242319
Initramfs Address : 0x60761f28
Initramfs Size    : 0xdbec9d/14412957
Compression Format: .mzip

Bootable image at @ ram:0x6044a5d8
Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x811060f0
Loading Linux kernel with entry point 0x811060f0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin

Reading bundle usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin...

Validating bundle usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin...
Installing bundle usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin...
Verifying bundle usbflash0:cat3k_caa-universalk9.SPA.03.06.00.E.152-2.E.bin...
Package cat3k_caa-base.SPA.03.06.00E.pkg is Digitally Signed
Package cat3k_caa-drivers.SPA.03.06.00E.pkg is Digitally Signed
Package cat3k_caa-infra.SPA.03.06.00E.pkg is Digitally Signed
Package cat3k_caa-iosd-universalk9.SPA.152-2.E.pkg is Digitally Signed
Package cat3k_caa-platform.SPA.03.06.00E.pkg is Digitally Signed
Package cat3k_caa-wcm.SPA.10.2.102.0.pkg is Digitally Signed
Preparing flash...
Syncing device...
./recovery: line 265: 1350 Segmentation fault      $FLASHUTIL_EXEC_IGNORE_CASE --command setvar --name BOOT --value flash:packages.conf 2>&1
Emergency Install failed - unable to set BOOT var
Restarting system.ÿ

Austin Sabio · ‎09-07-2017

Check InayathUlla's response.

https://supportforums.cisco.com/t5/lan-switching-and-routing/unable-to-change-boot-file-on-cisco-3850/td-p/2322258

Johnny Nguyen · ‎09-07-2017

thanks for your post when i followed it i got the same issue like this

FIPS(NGWC): Flash Key Check : Begin
FIPS(NGWC): Flash Key Check : End, Not Found, FIPS Mode Not Enabled
Sep 7 19:30:25 %IOSXE-3-PLATFORM: process sysmgr: Service [stackSep 7 19:30:26 %IOSXE-2-PLATFORM: process ng_dumper: Process stack-mgr: terminated abnormally.
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: ===================== Exception Data ====================
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Exception in 'stack-mgr' with Signal=SIGSEGV(11)
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Frame Pointer=0x7FD54490, Program Counter=0x2C86AAD8
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr: Thread ID=6325: -Traceback= 1#972af6ff8130dc1a2a2858bbb94cad79 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 ld:2B947000+AE60 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 crashinfo:2D7A0000+5A8 initcii:2D78C000+F28 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 c:2C950068+8960 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D78C000+0 initcii:2D7
Sep 7 19:30:27 %IOSXE-2-PLATFORM: process stack-mgr:

<Thu Sep 7 19:30:25 2017> Message from sysmgr: Reason Code:[2] Reset Reason:Service [stack-mgr] pid:[6325] terminated abnormally [11].
Service: stack-mgr
Description: Stack Manager
Executable: /tmp/sw/mount/cat3k_caa-platform.SPA.03.07.05E.pkg//usr/binos/bin/stack-mgr

Started at Thu Sep 7 19:29:25 2017 (747866 us)
Stopped at Thu Sep 7 19:30:25 2017 (917365 us)
Uptime: 1 minutes 0 seconds

Start type: SRV_OPTION_RESTART_STATELESS (23)
Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2)
Last heartbeat 0.00 secs ago

PID: 6325
Exit code: signal 11

CWD: /var/sysmgr/work

PID: 6325
UUID: 3006
get_local: failed: No such file or directory

/dev/fd/9: line 20: 8142 Segmentation fault (core dumped) flashutil -c setvar -N ABNORMAL_RESET_COUNT -V ${abnormal_reset_count}

can anyone help me ?

Austin Sabio · ‎09-08-2017

Try a very old stable image if that still doesnt work then open a cisco tac case- I would guess flash may be corrupted---but don't format flash before working with cisco tac it may void the warranty.

https://supportforums.cisco.com/t5/lan-switching-and-routing/3850-corrupt-flash-reformat/td-p/2914637