07-03-2025
06:01 AM
- last edited on
07-03-2025
06:40 AM
by
shaiksh
here Putty record , I have two switch same issue .
Booting...
Interface GE 0 link down***ERROR: PHY link is down
The "IP_ADDR" environment variable is not set.
Getting rest of image
Reading full image into memory...Check base package header ...: done = 16384
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 22301472
Bundle Image
--------------------------------------
Kernel Address : 0x53778384
Kernel Size : 0x34e9e1/3467745
Initramfs Address : 0x53ac6d65
Initramfs Size : 0x119d5bb/18470331
Compression Format: mzip
Bootable image at @ ram:0x53778384
Bootable image segment 0 address range [0x81100000, 0x81b8adc0] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 623
Loading Linux kernel with entry point 0x816902d0 ...
Bootloader: Done loading app on core_mask: 0xf
### Launching Linux Kernel (flags = 0x5)
Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1
Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...86376 bytes.
Front-end Microcode IMG MGR: Programming device 0...
% Front-end Microcode IMG MGR: address 0, Could not enable sys_poll
% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache
Both links down, not waiting for other switches
Switch number is 1
Jul 3 11:56:01.546 R0/0: %PMAN-3-PROCHOLDDOWN: The process platform_mgr has been helddown (rc 134)
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.5b, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Thu 02-Nov-17 11:07 by mcpre
Cisco IOS-XE software, Copyright (c) 2005-2017 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.
FIPS: Flash Key Check : Begin
FIPS: Flash Key Check : End, Not Found, FIPS Mode Not Enabled
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco WS-C3650-24PD (MIPS) processor (revision U0) with 865070K/6147K bytes of memory.
Processor board ID FDO2318F0GC
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
1611414K bytes of Flash at flash:.
0K bytes of at webui:.
Base Ethernet MAC Address : 2c:4f:52:62:22:00
Motherboard Assembly Number : 73-15894-07
Motherboard Serial Number : FDO23171VNT
Model Revision Number : U0
Motherboard Revision Number : A0
Model Number : WS-C3650-24PD
System Serial Number : FDO2318F0GC
Press RETURN to get started!
*Jul 3 11:56:38.257: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*Jul 3 11:56:42.821: dev_pluggable_optics_selftest attribute table internally inconsistent @ 0x125
*Jul 3 11:56:43.363: %SMI-5-CLIENT: Smart Install Client feature is enabled. It is recommended to disable the Smart Install feature when it is not actively used. To disable feature execute 'no vstack' in configuration mode
*Jul 3 11:56:43.461: %NBAR-6-CACHE_SYNC_INFO: Cache synchronization. Initialized.
*Jul 3 11:56:46.798: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jul 3 11:56:47.294: %LINK-3-UPDOWN: Interface Lsmpi18/0/3, changed state to up
*Jul 3 11:56:47.295: %LINK-3-UPDOWN: Interface EOBC18/0/1, changed state to up
*Jul 3 11:56:47.295: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
*Jul 3 11:56:47.295: %LINK-3-UPDOWN: Interface LIIN18/0/2, changed state to up
*Jul 3 11:56:47.647: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
Chassis 1 reloading, reason - Reload command
*Jul 3 11:56:47.714: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!
*Jul 3 11:56:47.745: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
*Jul 3 11:56:30.791: %STACKMGR-1-STACK_LINK_CHANGE:Switch 1 R0/0: stack_mgr: Stack port 1 on switch 1 is nocable
*Jul 3 11:56:30.792: %STACKMGR-1-STACK_LINK_CHANGE:Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is down
*Jul 3 11:56:30.792: %STACKMGR-1-STACK_LINK_CHANGE:Switch 1 R0/0: stack_mgr: Stack port 2 on switch 1 is nocable
*Jul 3 11:56:30.792: %STACKMGR-6-SWITCH_ADDED:Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*Jul 3 11:56:30.792: %STACKMGR-6-SWITCH_ADDED:Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*Jul 3 11:56:30.792: %STACKMGR-6-SWITCH_ADDED:Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
*Jul 3 11:56:48.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi18/0/3, changed state to up
*Jul 3 11:56:48.400: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC18/0/1, changed state to up
*Jul 3 11:56:48.400: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
*Jul 3 11:56:48.401: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN18/0/2, changed state to up
*Jul 3 11:56:50.509: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-vrf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Jul 3 11:56:51.924: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Jul 3 11:56:57.049: %SYS-6-STARTUP_CONFIG_IGNORED: System startup configuration is ignored based on the configuration register setting.
*Jul 3 11:56:59.791: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
*Jul 3 11:57:00.538: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
*Jul 3 11:57:02.322: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.5b, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Thu 02-Nov-17 11:07 by mcpre
*Jul 3 11:57:07.928: %PMAN-0-PROCFAILCRIT:Switch 1 R0/0: pvp.sh: A critical process platform_mgr has failed (rc 134)
*Jul 3 11:57:08.534: %PMAN-3-RELOAD_SYSTEM:Switch 1 R0/0: pvp.sh: Reloading: Peer switch is not standby ready. System will be reloaded
*Jul 3 11:57:08.564: %STACKMGR-1-RELOAD:Switch 1 R0/0: stack_mgr: Reloading due to reason Reload command Jul 3 11:57:18.501 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fp action requested
*Jul 3 11:57:18.538: %PMAN-5-EXITACTION:Switch 1 R0/0: pvp.sh: Process manager is exiting: reload fp action requestedJul 3 11:57:34.535 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: rp processes exit with reload switch code
Jul 3 11:57:52.470 R0/0: %PMAN-3-PROCESS_NOTIFICATION: System report /crashinfo/system-report_1_20250703-115743-UTC.tar.gz (size: 5739 KB) generated
octeon_wdt: WDT device closed unexpectedly. WDT will not stop!
reboot: Restarting system
07-03-2025 06:04 AM
I have installed last IOS 16.12.13 but no way same issue.
pleas your advice
07-03-2025 10:46 AM
hello. The continuous reboot cycle with "platform_mgr has failed (rc 134)" typically indicates either corrupted firmware or hardware issues. For ur ws-C3650-24pd/ switches u need to try and boot the golden bundle image in rommon mode, should be like this: #flash_init and then : boot flash:cat3k_caa-universalk9.SSA.03.07.05.E.152-3.E5.bin
hoping this works, but if it doesnt, u'll need to first completely reinstall IOS via rommon/ using usb ortftp. 2. replace the flash drive if corruption persists, and check power supply stabillity....
So yes,/ dev_pluggable_optics_selftest error, suggests possible hardware faults.
Try those, and let me know the outcome...
hope it helps G
-Enes
07-03-2025 10:51 AM
and also G, I just saw that someone had the same issue back on 2022: check the link, and hope it helps:
07-03-2025 12:37 PM
Hello @Essam Ahmed,
based on your output the root cause for the crash loop might be a failed microcode upgrade:
Front-end Microcode IMG MGR: found 1 microcode images for 1 device.
Image for front-end 0: /tmp/microcode_update/front_end/fe_type_8_1
Front-end Microcode IMG MGR: Preparing to program device microcode...
Front-end Microcode IMG MGR: Preparing to program device[0]...86376 bytes.
Front-end Microcode IMG MGR: Programming device 0...
% Front-end Microcode IMG MGR: address 0, Could not enable sys_poll
% Front-end Microcode IMG MGR: unable to program device: 0
% Front-end Microcode IMG MGR: malformed microcode cache
You can try a full power cycle to clear transient microcode issues.
A reload or warm boot is NOT enough!
You need to unplug all power cables, wait a few seconds and then recable and turn on the switch again.
If this does not help, then you need to open a TAC case and attach the crash report that was generated.
HTH!
07-03-2025 02:47 PM
@Essam Ahmed wrote:% Front-end Microcode IMG MGR: unable to program device: 0 % Front-end Microcode IMG MGR: malformed microcode cache
Contact Cisco TAC and organize and RMA.
It looks like someone interrupted the switch during a micro-code upgrade and corrupted the sector beyond repair.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide