Cisco 9300 - Cmand Errors

[CiscoReece]

Hello,

I'm trying to boot from usbflash0 (IOS 17.3.5)and flash:packages.conf but having the below errors.

%PMAN-3-PROCHOLDDOWN: R0/0: The process cmand has been helddown (rc 134)
%PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process cmand has failed (rc 134)
%PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Switch will be reloaded
Chassis 0 reloading, rrticipant detected
%PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fp action requested
%PMAN-5-EXITACTION: Rvp: Process manager is exiting: Critical process cmand fault on rp_0_0 (rc=134)

I have also tried an emergency-install which had come back as successful but I'm still seeing the above error

Starting system recovery (usbflash0:cat9k_iosxe.17.03.05.SPA.bin) ...

boot: attempting to boot from [sda9:cat9k-recovery.SSA.bin]
boot: reading file cat9k-recovery.SSA.bin

PLATFORM_TYPE C9300 speed 9600
Booting Recovery Image 16.6.3

Initiating Emergency Installation of bundle usbflash0:cat9k_iosxe.17.03.05.SPA.bin
Reading bundle usbflash0:cat9k_iosxe.17.03.05.SPA.bin...
Validating bundle usbflash0:cat9k_iosxe.17.03.05.SPA.bin...
Installing bundle usbflash0:cat9k_iosxe.17.03.05.SPA.bin....
Verifying bundle usbflash0:cat9k_iosxe.17.03.05.SPA.bin...
Package cat9k-cc_srdriver.17.03.05.SPA.pkg /temp//stage/cat9k-cc_srdriver.17.03.05.SPA.pkg is Digitally
Package cat9k-espbase.17.03.05.SPA.pkg /temp//stage/cat9k-espbase.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-guestshell.17.03.05.SPA.pkg /temp//stage/cat9k-guestshell.17.03.05.SPA.pkg is Digitally Si
Package cat9k-lni.17.03.05.SPA.pkg /temp//stage/cat9k-lni.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-rpbase.17.03.05.SPA.pkg /temp//stage/cat9k-rpbase.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-sipbase.17.03.05.SPA.pkg /temp//stage/cat9k-sipbase.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-sipspa.17.03.05.SPA.pkg /temp//stage/cat9k-sipspa.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-srdriver.17.03.05.SPA.pkg /temp//stage/cat9k-srdriver.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-webui.17.03.05.SPA.pkg /temp//stage/cat9k-webui.17.03.05.SPA.pkg is Digitally Signed
Package cat9k-wlc.17.03.05.SPA.pkg /temp//stage/cat9k-wlc.17.03.05.SPA.pkg is Digitally Signed
Package /cat9k-rpboot.17.03.05.SPA.pkg /temp//rpboot/cat9k-rpboot.17.03.05.SPA.pkg is Digitally Signed
Preparing flash....
Flash filesystem unmounted successfully /dev/sdb3
Syncing device....
Emergency Install success. Will reboot now
Initializing Hardware......

System Bootstrap, Version 17.3.2r, RELEASE SOFTWARE (P)
Compiled Tue 08/25/2020 23:46:12.85 by rel

Current ROMMON image : Primary
Last reset cause     : SoftwareReload
C9300-48P platform with 8388608 Kbytes of main memory

boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf

Does anyone have any suggestions please?

Thanks,

Reece

[balaji.bandi]

Looks for me BUG

[Leo Laohoo]

In ROMMON, post the complete output to the command "set".

[CiscoReece]

switch: set
ABNORMAL_RESET_COUNT=1
AUTOREBOOT_RESTORE=0
BAUD=9600
BOARDID=24660
BSI=0
BYPASS_STARTUP_CONFIG=1
DC_COPY=yes
LICENSE_BOOT_LEVEL=network-advantage+dna-advantage,all:ngwc;
MAC_ADDR=6C:6C:D3:B6:31:80
MANUAL_BOOT=no
MODEL_NUM=C9300-48P
MODEL_REVISION_NUM=A0
MOTHERBOARD_ASSEMBLY_NUM=73-18274-04
MOTHERBOARD_REVISION_NUM=A0
MOTHERBOARD_SERIAL_NUM=REMOVED
RANDOM_NUM=1830308837
RET_2_RCALTS=
RET_2_RTS=
ROMMON_AUTOBOOT_ATTEMPT=0
SWITCH_IGNORE_STARTUP_CFG=0
SWITCH_NUMBER=1
SYSTEM_SERIAL_NUM=REMOVED
TEMPLATE=access
VERSION_ID=V02

[Leo Laohoo]

---

@CiscoReece wrote:

%PMAN-3-PROCHOLDDOWN: R0/0: The process cmand has been helddown (rc 134)
%PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process cmand has failed (rc 134)
%PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Switch will be reloaded
Chassis 0 reloading, rrticipant detected
%PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fp action requested
%PMAN-5-EXITACTION: Rvp: Process manager is exiting: Critical process cmand fault on rp_0_0 (rc=134)​

---

This is beginning to look like CSCvr95394.  (Disregard the rc values.)

Console into the switch and boot.  Post the entire boot-up/crash process so we can see what is really going on with the switch.

Get ready to download different firmware version, like 16.9.X or 16.12.8 or 17.6.4.

[CiscoReece]

This is what happens when I boot from flash/USB

boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf
#
###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################



%PMAN-3-PROCHOLDDOWN: R0/0: The process cmand has been helddown (rc 134)
%PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process cmand has failed (rc 134)
%PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Switch will be reloaded

Chassis 0 reloading, reason - Non participant detected
                                                      Nov  9 15:40:04.753: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fp action requested
%PMAN-5-EXITACTION: Rvp: Process manager is exiting: Critical process cmand fault on rp_0_0 (rc=134)
%PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: Deleting old files in /crashinfo ... System report /crashinfo/RP_0-system-report_local_20221109-154006-Universal.tar.gz (size: 12208 KB) generated

Initializing Hardware......

System Bootstrap, Version 17.3.2r, RELEASE SOFTWARE (P)
Compiled Tue 08/25/2020 23:46:12.85 by rel

Current ROMMON image : Primary
Last reset cause     : SoftwareReload
C9300-48P platform with 8388608 Kbytes of main memory

[Leo Laohoo]

---

@CiscoReece wrote:

This is what happens when I boot from flash/USB

boot: attempting to boot from [flash:packages.conf]
boot: reading file packages.conf

---

What firmware is the switch booting now?

[CiscoReece]

It's currently booting 17.3.5

[Leo Laohoo · Accepted solution]

Download 17.6.4 into a USB thumb drive.  

In ROMMON, do the following:  boot usbflash0:IOS_filename.bin

[CiscoReece]

When I booted from 17.6.4 it actually worked! The bootloader and MCU upgraded and I can now log in! Thank you for helping me revive the switch!
I’ve forgot the enable password… What would be the best way to reset this password? Or just the whole switch? Emergency Install?

[Leo Laohoo]

Yes, emergency-install and start from scratch. 

[CiscoReece]

Have tried starting an emergency install a couple times and let it run for over an hour, it's getting as far as reading the recovery.bin file then hanging there.

[Leo Laohoo · Accepted solution]

Boot the firmware again using the USB flash drive. 

After it successfully boots up, copy the firmware from the USB flash to the switch flash.  Start the Install mode from there.

[kevin.philippe]

Hi Leo, 

I have the issue too, when booting on 16.12.07.

Issue came after firmware upgrade, switch always in boot loop with the following message 

Already tried an ermgency-install, but no success. Will try tomorrow a boot from usb with another version, f.ex 16.12.08

[Leo Laohoo]

CSCvr58845