Solved: ISR4431 reboot loop with %CPPHA-3-INITFAIL (0x43)

d_sergienko · ‎07-28-2025

Hi all,

One of the two ISR4431 started to boot into errors after power outage. I can enter ROMMON, able to boot many different IOSes (including external TFTP download via Gi0 interface), but all of them got stuck & reboot with the messages below (more or less verbose depending on IOS version).
I can't upgrade rommon because I'm unable to boot any IOS. ROMMON is 15.x, so only IOS-XE 3S is supported. Below is 3.15.01c. I'm not sure if this is a hardware fault or this is a software bug. I've tried to replace DIMMs for both CP and FFP taken from working unit, it did not help. Replacing PSU (both) are also has no effect.

I've seen similar threads here like this one, but no particular resolution found.

Appreciate your help.

Press RETURN to get started!

*Jul 28 11:36:50.411: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = esg Next reboot level = ipbasek9 and License = ipbasek9
*Jul 28 11:36:51.455: %ISR_THROUGHPUT-6-LEVEL: Throughput level has been set to 500000 kbps
*Jul 28 11:36:59.699: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jul 28 11:37:01.031: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Jul 28 11:37:01.031: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Jul 28 11:37:01.031: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Jul 28 11:37:01.031: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Jul 28 11:36:32.178: %CMRP-3-PFU_MISSING:cmand:  The platform does not detect a power supply in slot 1
*Jul 28 11:36:53.873: %CMLIB-6-THROUGHPUT_VALUE:cmand:  Throughput license found, throughput set to 500000 kbps
*Jul 28 11:36:55.575: %CPPHA-7-START:cpp_ha:  CPP 0 preparing ucode
*Jul 28 11:36:55.605: %CPPHA-7-START:cpp_ha:  CPP 0 startup init
*Jul 28 11:37:03.313: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Jul 28 11:37:03.365: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Jul 28 11:37:03.365: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Jul 28 11:37:03.366: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Jul 28 11:37:03.366: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Jul 28 11:37:03.366: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Jul 28 11:37:06.707: %LINK-5-CHANGED: Interface GigabitEthernet0/0/0, changed state to administratively down
*Jul 28 11:37:06.711: %LINK-5-CHANGED: Interface GigabitEthernet0/0/1, changed state to administratively down
*Jul 28 11:37:06.712: %LINK-5-CHANGED: Interface GigabitEthernet0/0/2, changed state to administratively down
*Jul 28 11:37:06.714: %LINK-5-CHANGED: Interface GigabitEthernet0/0/3, changed state to administratively down
*Jul 28 11:37:06.722: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Jul 28 11:37:07.708: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/0, changed state to down
*Jul 28 11:37:07.710: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/1, changed state to down
*Jul 28 11:37:07.713: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/2, changed state to down
*Jul 28 11:37:07.714: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0/3, changed state to down
*Jul 28 11:37:08.690: %CPPHA-3-INITFAIL:cpp_ha:  CPP 0 initialization failed - startup init (0x43)
*Jul 28 11:37:08.690: %CPPHA-3-INITFAIL:cpp_ha:  CPP 0 initialization failed - start CPP (0x43)
*Jul 28 11:37:08.690: %CPPHA-3-FAULT:cpp_ha:  CPP:0.0 desc:Platform Collection Error: Message STARTUP_INIT to client cpp_driver0 failed with error [Link has been severed] det:HA class:CLIENT_SW sev:FATAL id:2 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0
*Jul 28 11:37:08.692: %IOSXE-6-PLATFORM:cpp_ha: Shutting down CPP MDM while client(s) still connected
*Jul 28 11:37:09.000: %PMAN-3-PROCHOLDDOWN:pman.sh:  The process cpp_driver has been helddown (rc 255)
*Jul 28 11:37:09.003: %PMAN-3-PROCHOLDDOWN:pman.sh:  The process cpp_ha_top_level_server has been helddown (rc 69)
*Jul 28 11:37:09.265: %PMAN-0-PROCFAILCRIT:pvp.sh:  A critical process cpp_driver has failed (rc 255)
*Jul 28 11:37:13.321: %IOSXE-6-PLATFORM:cpp_cdm: Shutting down CPP MDM while client(s) still connected
*Jul 28 11:3Jul 28 11:37:34.163 R0/0: %PMAN-5-EXITACTION: Process manager is exiting: reload fru action requested
 

Initializing Hardware ...

Regards,

Dmitry

d_sergienko · ‎07-30-2025

Dear all,

Thanks @Leo Laohoo, M02@rt37 and @MHM Cisco World for your advices.

At this point I can affirm the problem is 100% hardware related. We found a way to revive the box: disassemble the router, pull out motherboard, connect power supply, fans and start without enclosure with caution of heatsinks overheating. We booted successfully about 10 times, no CPP errors any more. After that ROMMON and IOS were upgraded to 16.12(2r) and 17.9.5f accordingly.
I guess the root cause is MB is slightly bend, and it is causing bad contact problem somewhere inside MB or chips. For sure, cases like this are RMA only, otherwise you may have a huge PITA when put into production. It may act as a backup/testing hardware, and this is our case.

rtr#sh ver
Cisco IOS XE Software, Version 17.09.05f
Cisco IOS Software [Cupertino], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.9.5f, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2025 by Cisco Systems, Inc.
Compiled Sun 16-Feb-25 16:29 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2025 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: 16.12(2r)

rtr uptime is 5 minutes
Uptime for this control processor is 7 minutes
System returned to ROM by Reload Command
System image file is "bootflash:isr4400-universalk9.17.09.05f.SPA.bin"
Last reload reason: Reload Command



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.



Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite                 Suite Current         Type           Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9     None                  Smart License  None
securityk9
appxk9

AdvUCSuiteK9          None                  Smart License  None
uck9
cme-srst
cube


Technology Package License Information:

-----------------------------------------------------------------
Technology    Technology-package           Technology-package
              Current       Type           Next reboot
------------------------------------------------------------------
appxk9           None             Smart License    appxk9
uck9             None             Smart License    None
securityk9       None             Smart License    securityk9
ipbase           ipbasek9         Smart License    ipbasek9

The current throughput level is 500000 kbps


Smart Licensing Status: Smart Licensing Using Policy

cisco ISR4431/K9 (JUNO-1RU) processor with 7797598K/6147K bytes of memory.
Processor board ID FJC********
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
30584831K bytes of flash memory at bootflash:.

Configuration register is 0x2102

rtr#sh platform
Chassis type: ISR4431/K9

Slot      Type                State                 Insert time (ago)
--------- ------------------- --------------------- -----------------
0         ISR4431/K9          ok                    00:06:13
 0/0      ISR4431-X-4x1GE     ok                    00:04:47
R0        ISR4431/K9          ok, active            00:06:13
F0        ISR4431/K9          ok, active            00:06:13
P0        PWR-4430-AC         ok                    00:05:16
P1        Unknown             empty                 never
P2        ACS-4430-FANASSY    ok                    00:05:16

Slot      CPLD Version        Firmware Version
--------- ------------------- ---------------------------------------
0         15010638            16.12(2r)
R0        15010638            16.12(2r)
F0        15010638            16.12(2r)

rtr#

Rgds,
D

View solution in original post

M02@rt37 · ‎07-28-2025

Hello @d_sergienko

Thanks for all details. It's very likely a hardware fault on the motherboard, possibly the CPP/FPGA or flash controller. Since you can enter ROMmon and initiate TFTP boot, you are not completely bricked, but with CPP consistently failing, you’ll need to RMA the router if it's still under support—or consider it unrecoverable if not...

Best regards
.ı|ı.ı|ı. If This Helps, Please Rate .ı|ı.ı|ı.

d_sergienko · ‎07-28-2025

Hello M02@rt37

Thanks for your prompt reply.

Unfortunately, these boxes are out of support. At the same time, it will be sad just to give up and move this router to trash bin.

Any ideas how to diagnose CPP failures? We can leverage soldering if needed. Lack of information prevents us to make any significant changes with the motherboard.

MHM Cisco World · ‎07-28-2025

If I have time I will check this issue tonight.

I think we can solve it.

I am not so sure but I will try my best

MHM

d_sergienko · ‎07-28-2025

Thanks a lot, @MHM Cisco World
It will be great if we find a way to restore

MHM Cisco World · ‎07-28-2025

Did you try usb?

MHM

d_sergienko · ‎07-28-2025

Yes. USB is not detected in ROMMON, but is being detected (at least prints its proper size) during IOS boot.
As I unable to enter IOS prompt, can't move on with neither rommon upgrade, nor IOS upgrade higher than 3S train.

MHM Cisco World · ‎07-28-2025

What format of usb is it fat32?

MHM

d_sergienko · ‎07-28-2025

Yes. It was formatted inside the second ISR4431 and IOS was copied from that working box. File system is native.

But I'm wondering what is your idea? How USB flash can fix CPP error?

Leo Laohoo · ‎07-28-2025

*Jul 28 11:37:08.690: %CPPHA-3-INITFAIL:cpp_ha:  CPP 0 initialization failed - startup init (0x43)
*Jul 28 11:37:08.690: %CPPHA-3-INITFAIL:cpp_ha:  CPP 0 initialization failed - start CPP (0x43)

Console into the router and reboot. Post the entire boot-up process.

Something is missing.

d_sergienko · ‎07-28-2025

It works again after the night being powered off. I'm not sure the situation will remain stable. Probably there is a "bad contact" or similar intermittent failure. We will try to update ROMMON and load a newer IOS (17.x train) while it is alive. Will keep you updated. And below is the full boot log (successful one):

Initializing Hardware ...

System integrity status: 00000610
Key Sectors:(Primary,GOOD),(Backup,GOOD),(Revocation,GOOD)
Size of Primary = 2288 Backup = 2288 Revocation = 300

ROM:RSA Self Test Passed

Expected hash:
ddaf35a193617abacc417349ae204131
12e6fa4e89a97ea20a9eeee64b55d39a
2192992a274fc1a836ba3c23a3feebbd
454d4423643ce80e2a9ac94fa54ca49f

Obtained hash:
ddaf35a193617abacc417349ae204131
12e6fa4e89a97ea20a9eeee64b55d39a
2192992a274fc1a836ba3c23a3feebbd
454d4423643ce80e2a9ac94fa54ca49f
ROM:Sha512 Self Test Passed
Self Tests Latency: 376 msec
Rom image verified correctly


System Bootstrap, Version 15.4(2r)S, RELEASE SOFTWARE
Copyright (c) 1994-2014  by cisco Systems, Inc.

CPLD Version: 56 (MM/DD/YY): 01/06/15 Cisco ISR4431/K9 Slot:0

Current image running: Boot ROM0

Last reset cause: LocalSoft
Reading confreg 0xa142

Reading monitor variables from NVRAM
Enabling interrupts...done

Checking for PCIe device presence...done
Cisco ISR4431/K9 platform with 16777216 Kbytes of main memory

autoboot entry: NVRAM VALUES: bootconf: 0x2, autobootstate: 0
autobootcount: 0, autobootsptr: 0x0

no valid BOOT image found
Final autoboot attempt from default boot device...
File size is 0x0000001e
Located throughput_monitor_params
Image size 30 inode num 12, bks cnt 1 blk size 8*512
#
File is comprised of 1 fragments (100%)

Booting image with bootparam="root=/dev/ram rw console=ttyS2,9600n8 max_loop=64 pciehp.pciehp_force SR_BOOT=bootflash:/throughput_monitor_params"
Boot image size = 30 (0x1e) bytes


Image Base is: 0x00000000bced7018
Image Size is: 0x0000001e

Unsigned package found,  aborting ...
File size is 0x1beddda4
Located isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin
Image size 468573604 inode num 14, bks cnt 114398 blk size 8*512
###############################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
File is comprised of 117 fragments (0%)

Booting image with bootparam="root=/dev/ram rw console=ttyS2,9600n8 max_loop=64 pciehp.pciehp_force SR_BOOT=bootflash:/isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin"
Boot image size = 468573604 (0x1beddda4) bytes


Image Base is: 0x000000009fcf1018
Image Size is: 0x1beddda4

ROM:RSA Self Test Passed

Expected hash:
ddaf35a193617abacc417349ae204131
12e6fa4e89a97ea20a9eeee64b55d39a
2192992a274fc1a836ba3c23a3feebbd
454d4423643ce80e2a9ac94fa54ca49f

Obtained hash:
ddaf35a193617abacc417349ae204131
12e6fa4e89a97ea20a9eeee64b55d39a
2192992a274fc1a836ba3c23a3feebbd
454d4423643ce80e2a9ac94fa54ca49f
ROM:Sha512 Self Test Passed
Self Tests Latency: 377 msec

Package header rev 1 structure detected
Calculating SHA-1 hash...done
validate_package: SHA-1 hash:
        calculated c6953580:1b6cee2b:63b74a5b:eec0c0ef:863ea46c
        expected   c6953580:1b6cee2b:63b74a5b:eec0c0ef:863ea46c
Package type:30000, flags:0x0
Parsing package TLV info:
000: 00000009:0000001d:4b45595f:544c565f -         KEY_TLV_
010: 5041434b:4147455f:434f4d50:41544942 - PACKAGE_COMPATIB
020: 494c4954:59000000:00000009:0000000b - ILITY
030: 4652555f:52505f54:59504500:00000009 - FRU_RP_TYPE
040: 00000018:4b45595f:544c565f:5041434b -     KEY_TLV_PACK
050: 4147455f:424f4f54:41524348:00000009 - AGE_BOOTARCH
060: 0000000e:41524348:5f693638:365f5459 -     ARCH_i686_TY
070: 50450000:00000009:00000014:4b45595f - PE          KEY_
080: 544c565f:424f4152:445f434f:4d504154 - TLV_BOARD_COMPAT
090: 00000009:00000013:424f4152:445f6f76 -         BOARD_ov
0a0: 65726c6f:72645f54:59504500:00000009 - erlord_TYPE
0b0: 00000018:4b45595f:544c565f:43525950 -     KEY_TLV_CRYP
0c0: 544f5f4b:45595354:52494e47:00000009 - TO_KEYSTRING

TLV(0): T=9, L=29, V=KEY_TLV_PACKAGE_COMPATIBILITY
FRU Key detected
TLV(1): T=9, L=11, V=FRU_RP_TYPE
found package fru type FRU_RP_TYPE
TLV(2): T=9, L=24, V=KEY_TLV_PACKAGE_BOOTARCH
ARCH Key detected
TLV(3): T=9, L=14, V=ARCH_i686_TYPE
found package arch type ARCH_i686_TYPE
TLV(4): T=9, L=20, V=KEY_TLV_BOARD_COMPAT
TLV(5): T=9, L=19, V=BOARD_overlord_TYPE
TLV(6): T=9, L=24, V=KEY_TLV_CRYPTO_KEYSTRING
TLV(7): T=9, L=10, V=EnCrYpTiOn
TLV(8): T=9, L=11, V=CW_BEGIN=$$
TLV(9): T=9, L=19, V=CW_FAMILY=$isr4400$
TLV(10): T=9, L=64, V=CW_IMAGE=$isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin$
TLV(11): T=9, L=19, V=CW_VERSION=$3.15.1$
TLV(12): T=9, L=52, V=CW_DESCRIPTION=$Cisco IOS Software, IOS-XE Software$
TLV(13): T=9, L=9, V=CW_END=$$
found DIGISIGN TLV type 12 length = 392

RSA Signed RELEASE Image Signature Verification Successful.
Package Load Test Latency : 9870 msec
Image validated
%IOSXEBOOT-4-BOOT_SRC: (rp/0): mounting /boot/super.iso to /tmp/sw/isos

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706



Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(2)S1c, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Sat 15-Aug-15 21:40 by mcpre



Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco ISR4431/K9 (1RU) processor with 7809895K/6147K bytes of memory.
Processor board ID FJC1939D0NP
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16777216K bytes of physical memory.
30584831K bytes of flash memory at bootflash:.



Press RETURN to get started!


*Jan  1 17:14:30.036: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = esg Next reboot level = ipbasek9 and License = ipbasek9
*Jan  1 17:14:31.072: %ISR_THROUGHPUT-6-LEVEL: Throughput level has been set to 500000 kbps
*Jan  1 17:14:39.652: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jan  1 17:14:40.974: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Jan  1 17:14:40.974: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Jan  1 17:14:40.974: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Jan  1 17:14:40.975: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Jan  1 17:14:12.357: %CMRP-3-PFU_MISSING:cmand:  The platform does not detect a power supply in slot 0
*Jan  1 17:14:33.014: %CMLIB-6-THROUGHPUT_VALUE:cmand:  Throughput license found, throughput set to 500000 kbps
*Jan  1 17:14:34.760: %CPPHA-7-START:cpp_ha:  CPP 0 preparing ucode
*Jan  1 17:14:34.791: %CPPHA-7-START:cpp_ha:  CPP 0 startup init
*Jan  1 17:14:43.312: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Jan  1 17:14:43.363: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Jan  1 17:14:43.364: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Jan  1 17:14:43.364: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Jan  1 17:14:43.364: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Jan  1 17:14:43.364: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Jan  1 17:14:44.659: %SYS-6-STARTUP_CONFIG_IGNORED: System startup configuration is ignored based on the configuration register setting.
*Jan  1 17:14:43.878: %CPPHA-7-START:cpp_ha:  CPP 0 running init
*Jan  1 17:14:44.196: %CPPHA-7-READY:cpp_ha:  CPP 0 loading and initialization complete
*Jan  1 17:14:44.335: %IOSXE-6-PLATFORM:cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER
*Jan  1 17:14:44.670: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/0, interfaces disabled
*Jan  1 17:14:44.674: %SPA_OIR-6-OFFLINECARD: SPA (ISR4431-X-4x1GE) offline in subslot 0/0
*Jan  1 17:14:44.677: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
*Jan  1 17:14:44.677: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0
*Jan  1 17:14:44.678: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
*Jan  1 17:14:44.678: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 0
*Jan  1 17:14:44.686: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/0
*Jan  1 17:14:44.713: %SPA-3-ENVMON_NOT_MONITORED:iomd:  Environmental monitoring is not enabled for ISR4431-X-4x1GE[0/0]
*Jan  1 17:14:44.785: %SYS-5-RESTART: System restarted --
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(2)S1c, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Sat 15-Aug-15 21:40 by mcpre
*Jan  1 17:14:46.945: %SPA_OIR-6-ONLINECARD: SPA (ISR4431-X-4x1GE) online in subslot 0/0
*Jan  1 17:14:47.819: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Jan  1 17:14:48.834: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Jan  1 17:14:48.834: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/0, changed state to down
*Jan  1 17:14:48.885: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/1, changed state to down
*Jan  1 17:14:48.887: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/2, changed state to down
*Jan  1 17:14:48.938: %LINK-3-UPDOWN: Interface GigabitEthernet0/0/3, changed state to down
*Jan  1 17:15:00.549: %SSH-5-ENABLED: SSH 1.99 has been enabled
Router>en
Router#

Leo Laohoo · ‎07-29-2025

@d_sergienko wrote:
Reading confreg 0xa142

Da fuq?

@d_sergienko wrote:

Booting image with bootparam="root=/dev/ram rw console=ttyS2,9600n8 max_loop=64 pciehp.pciehp_force SR_BOOT=bootflash:/isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin"

Post the complete output to the following commands:

dir
sh boot

d_sergienko · ‎07-29-2025

@Leo Laohoo wrote:
@d_sergienko wrote:

Reading confreg 0xa142

Da fuq?

0x2142 | 0x8000 = 0xA142 (enable extra boot debug).

It has successfully booted 2 times subsequently and then the problem appeared again.

Posting "dir bootflash:" when it worked today:

Router#dir
Directory of bootflash:/
   11  drwx             4096  Jul 24 2025 13:51:53 +00:00  lost+found
1275457  drwx             4096   Jan 1 2010 01:42:16 +00:00  core
1520737  drwx             4096   Jan 1 2010 17:33:16 +00:00  .prst_sync
3041473  drwx             4096  Sep 23 2015 14:46:03 +00:00  .rollback_timer
3728257  drwx            98304   Jan 1 2010 17:33:26 +00:00  tracelogs
2599969  drwx             4096  Sep 23 2015 14:46:14 +00:00  .installer
2191169  drwx             4096  Sep 23 2015 14:46:16 +00:00  virtual-instance
   12  -rw-               30   Jan 1 2010 17:33:25 +00:00  throughput_monitor_params
   14  -rw-        468573604  Sep 23 2015 14:59:37 +00:00  isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin
 
30827036672 bytes total (28784476160 bytes free)

Leo Laohoo · ‎07-29-2025

@d_sergienko wrote:

Router#dir
Directory of bootflash:/
   11  drwx             4096  Jul 24 2025 13:51:53 +00:00  lost+found
1275457  drwx             4096   Jan 1 2010 01:42:16 +00:00  core
1520737  drwx             4096   Jan 1 2010 17:33:16 +00:00  .prst_sync
3041473  drwx             4096  Sep 23 2015 14:46:03 +00:00  .rollback_timer
3728257  drwx            98304   Jan 1 2010 17:33:26 +00:00  tracelogs
2599969  drwx             4096  Sep 23 2015 14:46:14 +00:00  .installer
2191169  drwx             4096  Sep 23 2015 14:46:16 +00:00  virtual-instance
   12  -rw-               30   Jan 1 2010 17:33:25 +00:00  throughput_monitor_params
   14  -rw-        468573604  Sep 23 2015 14:59:37 +00:00  isr4400-universalk9.03.15.01c.S.155-2.S1c-std.SPA.bin
 
30827036672 bytes total (28784476160 bytes free)

Please post the complete output to the following commands:

dir bootflash:core
dir bootflash:tracelogs

d_sergienko · ‎07-30-2025

@Leo Laohoo wrote:
Please post the complete output to the following commands:

dir bootflash:core
dir bootflash:tracelogs

Could not do this on a live system, so switched eMMC bootflash onto working box and booted there.
bootflash:core :

30-Jul-25  12:06                 1 .callhome
24-Jul-25  16:52    <DIR>          modules
01-Jan-10  04:42         1,562,996 RP_0_cman_fp_12613_1262310134.core.gz

bootflash:tracelogs:

30-Jul-25  11:53               120 bexecute.sh-16372_R0-0.log.gz
30-Jul-25  11:53               120 bexecute.sh-16667_R0-0.log.gz
30-Jul-25  11:56               119 bexecute.sh-20838_R0-0.log.gz
30-Jul-25  11:56               120 bexecute.sh-22130_R0-0.log.gz
28-Jul-25  13:56               160 binos_log_R0-0.log.10555.20250728105646.gz
28-Jul-25  14:04               160 binos_log_R0-0.log.10557.20250728110430.gz
25-Jul-25  16:03               162 binos_log_R0-0.log.10565.20250725130343.gz
28-Jul-25  14:00               161 binos_log_R0-0.log.10566.20250728110038.gz
28-Jul-25  14:33               161 binos_log_R0-0.log.10570.20250728113330.gz
28-Jul-25  14:11               161 binos_log_R0-0.log.10571.20250728111117.gz
25-Jul-25  18:08               161 binos_log_R0-0.log.10572.20250725150820.gz
25-Jul-25  18:25               161 binos_log_R0-0.log.10573.20250725152511.gz
28-Jul-25  13:52               160 binos_log_R0-0.log.10573.20250728105258.gz
28-Jul-25  14:37               161 binos_log_R0-0.log.10576.20250728113735.gz
28-Jul-25  15:07               161 binos_log_R0-0.log.10579.20250728120737.gz
25-Jul-25  15:59               161 binos_log_R0-0.log.10591.20250725125951.gz
28-Jul-25  14:45               161 binos_log_R0-0.log.10595.20250728114556.gz
30-Jul-25  12:08            16,662 boottime
30-Jul-25  11:53               115 brelay-16368_R0-0.log.gz
30-Jul-25  11:53               114 brelay-16663_R0-0.log.gz
30-Jul-25  11:56               114 brelay-20834_R0-0.log.gz
30-Jul-25  11:56               115 brelay-22126_R0-0.log.gz
25-Jul-25  16:03             1,492 bsm_pmanlog_R0-0.log.22461.20250725130342.gz
28-Jul-25  14:00             1,482 bsm_pmanlog_R0-0.log.22473.20250728110036.gz
28-Jul-25  13:56             1,491 bsm_pmanlog_R0-0.log.22474.20250728105645.gz
28-Jul-25  14:04             1,491 bsm_pmanlog_R0-0.log.22479.20250728110429.gz
28-Jul-25  14:33             1,496 bsm_pmanlog_R0-0.log.22490.20250728113329.gz
25-Jul-25  18:08             1,481 bsm_pmanlog_R0-0.log.22491.20250725150818.gz
28-Jul-25  14:11             1,487 bsm_pmanlog_R0-0.log.22491.20250728111116.gz
28-Jul-25  14:37             1,489 bsm_pmanlog_R0-0.log.22493.20250728113734.gz
25-Jul-25  18:25             1,488 bsm_pmanlog_R0-0.log.22495.20250725152510.gz
28-Jul-25  13:52             1,474 bsm_pmanlog_R0-0.log.22499.20250728105257.gz
28-Jul-25  15:07             1,492 bsm_pmanlog_R0-0.log.22499.20250728120736.gz
28-Jul-25  14:45             1,478 bsm_pmanlog_R0-0.log.22518.20250728114555.gz
28-Jul-25  14:00             1,834 bsm_R0-0.log.23135.20250728110036.gz
28-Jul-25  13:56             1,843 bsm_R0-0.log.23141.20250728105645.gz
25-Jul-25  16:03             1,835 bsm_R0-0.log.23144.20250725130342.gz
28-Jul-25  14:11             1,831 bsm_R0-0.log.23154.20250728111116.gz
28-Jul-25  14:33             1,821 bsm_R0-0.log.23156.20250728113329.gz
25-Jul-25  18:25             1,466 bsm_R0-0.log.23167.20250725152510.gz
28-Jul-25  13:52             1,832 bsm_R0-0.log.23177.20250728105257.gz
28-Jul-25  14:45             1,828 bsm_R0-0.log.23182.20250728114555.gz
28-Jul-25  14:04             1,913 bsm_R0-0.log.23184.20250728110429.gz
28-Jul-25  15:07             1,826 bsm_R0-0.log.23185.20250728120736.gz
25-Jul-25  18:08             1,483 bsm_R0-0.log.23186.20250725150818.gz
25-Jul-25  15:59             1,916 bsm_R0-0.log.23188.20250725125950.gz
......
28-Jul-25  14:11             1,994 vman_R0-0.log.30846.20250728111117.gz
28-Jul-25  13:56             2,596 vman_R0-0.log.30848.20250728105646.gz
25-Jul-25  18:25             3,016 vman_R0-0.log.30876.20250725152511.gz
28-Jul-25  13:52             2,600 vman_R0-0.log.30886.20250728105258.gz
28-Jul-25  14:37             2,628 vman_R0-0.log.30908.20250728113735.gz
28-Jul-25  15:07             2,000 vman_R0-0.log.30924.20250728120737.gz
28-Jul-25  14:33             2,550 vman_R0-0.log.30983.20250728113330.gz
28-Jul-25  14:04             2,606 vman_R0-0.log.32275.20250728110430.gz
             875 File(s)      1,909,403 bytes

The most interesting I've found inside:

cman_fp_F0_0.log

07/28 12:06:57.227 [btrace]: [1956]: (note): Set level for module [95] [cpp-ha] to [5]                                                                                                                           07/28 12:06:57.227 [btrace]: [1956]: (note): Set level for module [94] [buginf] to [5]                                                                                                                           07/28 12:06:57.227 [buginf]: [1956]: (debug): FP Board Type: OVLD CPU type 2                                                                                                                                                                                                                                                                                                                                                      07/28 12:06:57.228 [btrace]: [1956]: (note): Set level for module [93] [cpp-oslib] to [5]                                                                                                                        07/28 12:06:57.229 [bipc]: [1956]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_ha_mgr_socket                                                                                                        07/28 12:06:57.229 [bipc]: [1956]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_ha_mgr_socket                                                                                                        07/28 12:06:58.488 [cman_fp]: [1956]: (ERR): failed to dispatch message 0: Function not implemented                                                                                                              07/28 12:06:58.488 [cman_fp]: [1956]: (ERR): failed to dispatch message 1: Function not implemented                                                                                                              07/28 12:06:58.489 [cman_fp]: [1956]: (ERR): failed to dispatch message 0: Function not implemented                                                                                                              07/28 12:06:58.489 [cman_fp]: [1956]: (ERR): failed to dispatch message 0: Function not implemented                                                                                                              07/28 12:06:58.489 [cman_fp]: [1956]: (ERR): failed to dispatch message 1: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 2: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 3: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 4: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 5: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 6: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 7: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 8: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 9: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 10: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 11: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 12: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 13: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 14: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 15: Function not implemented                                                                                                             07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 0: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 1: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 2: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 3: Function not implemented                                                                                                              07/28 12:06:58.490 [cman_fp]: [1956]: (ERR): failed to dispatch message 4: Function not implemented                                                                                                              07/28 12:07:10.554 [cman_fp]: [1956]: (ERR): CPP request 1 failed = Link has been severed                                                                                                                        07/28 12:07:10.594 [cpp-oslib]: [1956]: (warn): smc_msg_read_internal: Conn 0x84e9958 has terminated rc=32 - cleaning up                                                                                         07/28 12:07:10.594 [cpp-ha]: [1956]: (warn): CMAN-FP HA channel error detected

cpp_ha_F0-0.log

07/28 12:06:57.229 [bipc]: [3076]: (note): Received a connection from client for path /tmp/fp/lipc/cpp_ha_mgr_socket                                                                                             07/28 12:06:57.230 [bipc]: [3076]: (note): Received a connection from client for path /tmp/fp/lipc/cpp_ha_mgr_socket                                                                                             07/28 12:06:57.242 [buginf]: [3076]: (debug): cpp_dev_info:                                                                                                                                                        Page Size = 0x0000000000001000                                                                                                                                                                                   Start Phy Addr  = 0x421FB83C0                                                                                                                                                                                    End Phy Addr    = 0x421FB8468                                                                                                                                                                                    Start Map Addr  = 0x421FB8000                                                                                                                                                                                    Next Page Start = 0x421FB9000                                                                                                                                                                                    Map Length      = 0x1000                                                                                                                                                                                         Start Offset    = 0x3C0                                                                                                                                                                                                                                                                                                                                                                                                         07/28 12:06:57.242 [buginf]: [3076]: (debug): cpp_drv_cpp_dev_info_init: Set access error threshold to 1                                                                                                                                                                                                                                                                                                                          07/28 12:06:57.243 [btrace]: [3076]: (note): Set level for module [102] [i95] to [5]                                                                                                                             07/28 12:06:57.261 [btrace]: [3076]: (note): Set level for module [101] [devobj] to [5]                                                                                                                          07/28 12:06:57.287 [cpp-ha]: [3076]: (warn): cpp_ha_oor_svr_rm_reg_msg_hdlr: Multiple RM registration for same resource  bitmap=0x80                                                                             07/28 12:06:57.290 [cpp-ha]: [3076]: (warn): cpp_ha_oor_svr_rm_reg_msg_hdlr: Multiple RM registration for same resource  bitmap=0x80                                                                             07/28 12:06:57.379 [buginf]: [3076]: (debug): cpp_dal_get_access_key_cmn: Overriding default access error threshold of 1 with value 30                                                                                                                                                                                                                                                                                            07/28 12:06:57.379 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_cp_socket                                                                                                            07/28 12:06:57.379 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_cp_socket                                                                                                            07/28 12:06:57.379 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_drv0_socket                                                                                                          07/28 12:06:57.380 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_drv_ipc0_socket                                                                                                      07/28 12:06:57.380 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_drv_ipc0_socket                                                                                                      07/28 12:06:57.380 [btrace]: [3076]: (note): Set level for module [100] [errmsg] to [5]                                                                                                                          07/28 12:06:57.411 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_cp_socket                                                                                                            07/28 12:06:57.411 [bipc]: [3076]: (note): Successfuly connected to server /tmp/fp/lipc/cpp_cp_socket                                                                                                            07/28 12:07:10.553 [cpp-oslib]: [3076]: (warn): smc_msg_read_internal: Conn 0x7c9250 has terminated rc=32 - cleaning up                                                                                          07/28 12:07:10.554 [cpp-oslib]: [3076]: (warn): smc_msg_read_internal: Conn 0x920da0 has terminated rc=32 - cleaning up                                                                                          07/28 12:07:10.554 [cpp-oslib]: [3076]: (warn): smc_msg_read_internal: Conn 0x7dd440 has terminated rc=32 - cleaning up                                                                                          07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack_cmn: CPP HA Request state=Send Request Descend tid=0xa00001 failed                                                                           07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): ERROR: Message STARTUP_INIT to client cpp_driver0 failed with error [Link has been severed]                                                                         07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack: received non EOK from tid(0x800003) result(0x43)                                                                                            07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack_cmn: CPP HA start CPP state=Startup init tid=0x800003 failed                                                                                 07/28 12:07:10.554 [errmsg]: [3076]: (ERR): %CPPHA-3-INITFAIL: CPP 0 initialization failed - startup init (0x43)                                                                                                 07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): CPP HA start CPP: CPP startup init failed for CPP 0 - Link has been severed                                                                                         07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack: received non EOK from tid(0x700001) result(0x43)                                                                                            07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack_cmn: CPP HA Startup CPPs state=(null) tid=0x700001 failed                                                                                    07/28 12:07:10.554 [errmsg]: [3076]: (ERR): %CPPHA-3-INITFAIL: CPP 0 initialization failed - start CPP (0x43)                                                                                                    07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): CPP HA Startup CPPs: CPP start failed for CPP 0 - Link has been severed                                                                                             07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack: received non EOK from tid(0x400003) result(0x43)                                                                                            07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_tm_transaction_ack_cmn: CPP HA sysinit/start CPPs state=Start CPPs tid=0x400003 failed                                                                          07/28 12:07:10.554 [errmsg]: [3076]: (ERR): %CPPHA-3-FAULT: CPP:0.0 desc:Platform Collection Error: Message STARTUP_INIT to client cpp_driver0 failed with error [Link has been severed] det:HA class:CLIENT_SW sev:FATAL id:2 cppstate:RUNNING res:UNKNOWN flags:0x0 cdmflags:0x0                                                                                                                                                07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_ha_platform_send_request_a: no platform entity registered for request 9                                                                                         07/28 12:07:10.554 [cpp-ha]: [3076]: (warn): cpp_ha_platform_send_request_a: no platform entity registered for request 8                                                                                         07/28 12:07:10.555 [cpp-ha]: [3076]: (warn): cpp_ha_plat_restart_msg_queue_send: Send event PLATFORM_RESTART_EVENT (29) failed - 'Subsystem(84)' detected the 'resource not available' condition 'Code(2)': No such file or directory

All the rest looks like the result of these two traces.