WS-C3650-24PS-S can not boot to IOS in rommon mode

bachlx · ‎10-30-2023

<html>
I have 2 switches running stack, I factory reset both to use a single run switch

But can not boot to IOS in rommon mode

This is my logs:

Press CTRL-A Z for help on special keys

Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 30894576
Bundle Image
--------------------------------------
Kernel Address : 0x5342f3b4
Kernel Size : 0x3684e0/3572960
Initramfs Address : 0x53797894
Initramfs Size : 0x19b595c/26958172
Compression Format: mzip

Bootable image at @ ram:0x5342f3b4
Bootable image segment 0 address range [0x81100000, 0x81c14630] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816e4a80 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

May 23 00:54:22.020: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task boothelper_autoupgrade.service ( exit-code )
MM [0] MCU version 21 sw ver 0

Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1 update needed: yes

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0], index=0 ...88464 bytes.
Front-end Microcode IMG MGR: Programming device 0...
% Front-end Microcode IMG MGR: address 0, Could not enable sys_poll

% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache
Both links down, not waiting for other switches
Switch number is 1

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:33 by mcpre

PLEASE READ THE FOLLOWING TERMS CAREFULLY. INSTALLING THE LICENSE OR
LICENSE KEY PROVIDED FOR ANY CISCO SOFTWARE PRODUCT, PRODUCT FEATURE,
AND/OR SUBSEQUENTLY PROVIDED SOFTWARE FEATURES (COLLECTIVELY, THE
"SOFTWARE"), AND/OR USING SUCH SOFTWARE CONSTITUTES YOUR FULL
ACCEPTANCE OF THE FOLLOWING TERMS. YOU MUST NOT PROCEED FURTHER IF YOU
ARE NOT WILLING TO BE BOUND BY ALL THE TERMS SET FORTH HEREIN.

Your use of the Software is subject to the Cisco End User License Agreement
(EULA) and any relevant supplemental terms (SEULA) found at
http://www.cisco.com/c/en/us/about/legal/cloud-and-software/software-terms.html.

You hereby acknowledge and agree that certain Software and/or features are
licensed for a particular term, that the license to such Software and/or
features is valid only for the applicable term and that such Software and/or
features may be shut down or otherwise terminated by Cisco after expiration
of the applicable license term (e.g., 90-day trial period). Cisco reserves
the right to terminate any such Software feature electronically or by any
other means available. While Cisco may provide alerts, it is your sole
responsibility to monitor your usage of any such term Software feature to
ensure that your systems and networks are prepared for a shutdown of the
Software feature.

May 23 00:56:54.168: %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process platform_mgr has been helddown (rc 134)

FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled
cisco WS-C3650-24PS (MIPS) processor (revision V0) with 832441K/6147K bytes of memory.
Processor board ID FDO2323F16P
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
1611414K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : c4:f7:d5:45:9b:80
Motherboard Assembly Number : 73-15899-07
Motherboard Serial Number : FDO23230CH6
Model Revision Number : V0
Motherboard Revision Number : A0
Model Number : WS-C3650-24PS
System Serial Number : FDO2323F16P

Chassis 1 reloading, reason - Reload command

Press RETURN to get started!

*May 23 00:56:51.231: Rommon debug: debugFlagsStr[1], flags[0x1]
*May 23 00:56:53.008: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is not allowed
*May 23 00:56:56.168: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*May 23 00:56:56.169: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled
*May 23 00:56:56.171: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is not allowed
*May 23 00:56:56.201: %SMART_LIC-5-EVAL_START: Entering evaluation period
*May 23 00:57:09.272: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*May 23 00:57:10.912: %LINK-3-UPDOWN: Interface Lsmpi18/3, changed state to up
*May 23 00:57:10.920: %LINK-3-UPDOWN: Interface EOBC18/1, changed state to up
*May 23 00:57:10.922: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*May 23 00:57:10.922: %LINK-3-UPDOWN: Interface LIIN18/2, changed state to up
*May 23 00:57:11.618: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!
*May 23 00:57:11.630: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could notP
*May 23 00:54:21.976: %BOOT-3-SYSD_STARTFAIL: Switch 1 R0/0: root: Failed to launch boot task boothelper_autoupgrade.service ( exit-code )
*May 23 00:56:54.032: %PMAN-3-PROCHOLDDOWN: Switch 1 R0/0: pman: The process platform_mgr has been helddown (rc 134)
*May 23 00:56:54.948: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is nocable
*May 23 00:56:54.948: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is down
*May 23 00:56:54.948: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is nocable
*May 23 00:56:54.948: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*May 23 00:56:54.948: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*May 23 00:56:54.948: %STACKMGR-6-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*May 23 00:56:54.948: %STACKMGR-6-ACTIVE_ELECTED: Switch 1 R0/0: stack_mgr: Switch 1 has been elected ACTIVE.
*May 23 00:57:11.643: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could notP
*May 23 00:57:11.656: %SMART_LIC-6-HA_ROLE_CHANGED: Smart Agent HA role changed to Active.
*May 23 00:57:11.971: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi18/3, changed state to up
*May 23 00:57:11.971: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC18/1, changed state to up
*May 23 00:57:11.972: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*May 23 00:57:11.972: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN18/2, changed state to up
*May 23 00:57:14.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*May 23 00:57:14.324: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-vrf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E01
*May 23 00:57:16.127: %SYS-6-STARTUP_CONFIG_IGNORED: System startup configuration is ignored based on the configuration register setting.
*May 23 00:57:16.133: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*May 23 00:57:17.662: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
*May 23 00:57:18.118: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:33 by mcpre
*May 23 00:57:20.107: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
*May 23 00:57:19.999: %PMAN-0-PROCFAILCRIT: Switch 1 R0/0: pvp: A critical process platform_mgr has failed (rc 134)
*May 23 00:57:20.425: %PMAN-3-RELOAD_SYSTEM: Switch 1 R0/0: pvp: Reloading: Peer switch is not standby ready. System will be reloaded
*May 23 00:57:20.512: %STACKMGR-1-RELOAD: Switch 1 R0/0: stack_mgr: Reloading due to reason Reload command
*May 23 00:57:22.924: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*May 23 00:57:23.726: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)May 23 00:57:24.999: %PMAN-5-EXITACTION: F0d

*May 23 00:57:23.924: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*May 23 00:57:24.923: %PMAN-5-EXITACTION: Switch 1 R0/0: pvp: Process manager is exiting: reload fp action requested
*May 23 00:57:29.362: %PKI-2-NON_AUTHORITATIVE_CLOCK: PKI functions can not be initialized until an authoritative time source, like NTP, can e
May 23 00:57:43.166: %PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: System report /flash/system-report_1_20190523-005740-Universal.tar.gz (size: 29d

octeon_wdt: WDT device closed unexpectedly. WDT will no+-----------------------------+
reboot: Restarting system | |
| Cannot open /dev/ttyUSB0! |
| |
+-----------------------------+
Booting...
Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 30894576
Bundle Image
--------------------------------------
Kernel Address : 0x5342f3b4
Kernel Size : 0x3684e0/3572960
Initramfs Address : 0x53797894
Initramfs Size : 0x19b595c/26958172
Compression Format: mzip

Bootable image at @ ram:0x5342f3b4
Bootable image segment 0 address range [0x81100000, 0x81c14630] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816e4a80 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

After connect Stack

Booting...
Interface GE 0 link down***ERROR: PHY link is down

Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 30894576
Bundle Image
--------------------------------------
Kernel Address : 0x5342f3b4
Kernel Size : 0x3684e0/3572960
Initramfs Address : 0x53797894
Initramfs Size : 0x19b595c/26958172
Compression Format: mzip

Bootable image at @ ram:0x5342f3b4
Bootable image segment 0 address range [0x81100000, 0x81c14630] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816e4a80 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

May 23 00:54:15.028: %BOOT-3-SYSD_STARTFAIL: R0/0: Failed to launch boot task boothelper_autoupgrade.service ( exit-code )
MM [0] MCU version 21 sw ver 0

Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1 update needed: yes

Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0], index=0 ...88464 bytes.
Front-end Microcode IMG MGR: Programming device 0...
% Front-end Microcode IMG MGR: address 0, Could not enable sys_poll

+-----------------------------+
% Front-end Microcode IMG MGR: unable to program device:| |
% Front-end Microcode IMG MGR: malformed microcode cache| Cannot open /dev/ttyUSB0! |
| |
+-----------------------------+

Waiting for 120 seconds for other switches to boot
##########

#
#####May 23 00:56:47.453: %PMAN-3-PROCHOLDDOWN: R0/0: pman: The process platform_mgr has been helddown (rc 134)
May 23 00:56:47.693: %PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process platform_mgr has failed (rc 134)
May 23 00:56:47.963: %PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: Switch will be reloaded

Chassis 1 reloading, reason - Non participant detected
May 23 00:56:52.519: %PMAN-5-EXITACTION: F0/0: pvp: Process manager is exiting: reload fp action requested

Switch rebooted continously for atleast 5 times. Disabling autoreboot
May 23 00:57:00.745: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: rp processes exit with reload switch code
May 23 00:57:08.122: %PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: System report /flash/system-report_local_20190523-005704-Universal.tar.gz (sized

��/kSw{��w��ww��Yw�]��ߺ��ﺻ��

Booting...

Please help me how to boot into the switch's OS, Thank so much.

</html>

marce1000 · ‎10-31-2023

>...I have 2 switches running stack, I factory reset both to use a single run switch
?? I would expect that the switches must have a minimal configuration first before a stack can be formed, break the stack and check if they can boot (standalone)

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '

bachlx · ‎10-31-2023

I broke the stack and factory reset both switches, currently it cannot be booted (standalone)

marce1000 · ‎10-31-2023

- FYI : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvc03655

M.

-- Each morning when I wake up and look into the mirror I always say ' Why am I so brilliant ? '
When the mirror will then always repond to me with ' The only thing that exceeds your brilliance is your beauty! '